t. haiden, m. janousek, p. bauer, j. bidlot, m. dahoui, l ... · pdf fileappropriate non-...
TRANSCRIPT
765
Evaluation of ECMWF forecasts,
including 2014-2015 upgrades
T. Haiden, M. Janousek, P. Bauer, J. Bidlot, M. Dahoui, L. Ferranti, F. Prates,
D.S. Richardson and F. Vitart
Research and Forecast Department
November 2015
Series: ECMWF Technical Memoranda A full list of ECMWF Publications can be found on our web site under:
http://www.ecmwf.int/en/research/publications
Contact: [email protected] © Copyright 2015 European Centre for Medium Range Weather Forecasts Shinfield Park, Reading, Berkshire RG2 9AX, England Literary and scientific copyrights belong to ECMWF and are reserved in all countries. This publication is not to be reprinted or translated in whole or in part without the written permission of the Director. Appropriate non-commercial use will normally be granted under the condition that reference is made to ECMWF. The information within this publication is given in good faith and considered to be true, but ECMWF accepts no liability for error, omission and for loss or damage arising from its use.
EvaluationofECMWFforecastsincluding2014‐2015upgrades
TechnicalMemorandumNo.765 1
1. IntroductionRecentchangestotheECMWFforecastingsystemaresummarisedinsection2.VerificationresultsoftheECMWFmedium‐rangeupper‐airforecastsarepresentedinsection3,including,whereavailable,acomparisonofECMWF’sforecastperformancewiththatofotherglobalforecastingcentres.Section4presentstheverificationofECMWFforecastsofweatherparametersandoceanwaves,whilesevereweatherisaddressedinsection5.Finally,section6discussestheperformanceofmonthlyandseasonalforecastproducts.
Atits42ndSession(October2010),theTechnicalAdvisoryCommitteeendorsedasetoftwoprimaryandfoursupplementaryheadlinescorestomonitortrendsinoverallperformanceoftheoperationalforecastingsystem.Theseheadlinescoresareincludedinthecurrentreport.Asinpreviousreportsawiderangeofcomplementaryverificationresultsisincludedand,toaidcomparisonfromyeartoyear,thesetofadditionalverificationscoresshownhereismainlyconsistentwiththatofpreviousyears(ECMWFTech.Memos.346,414,432,463,501,504,547,578,606,635,654,688,710,742).Ashorttechnicalnotedescribingthescoresusedinthisreportisgivenintheannextothisdocument.
VerificationpageshavemostlybeenmovedtothenewECMWFwebsiteandareregularlyupdated.Theyareaccessibleatthefollowingaddress:
www.ecmwf.int/en/forecasts/charts
bychoosing‘Verification’undertheheader‘MediumRange’
(medium‐rangeandoceanwaves)
bychoosing‘Verification’undertheheader‘ExtendedRange’
(monthly)
bychoosing‘Verification’and‘Seasonalforecasts‘undertheheader‘LongRange’
(seasonal)
2. ChangestotheECMWFforecastingsystemInNovember2014,ECMWFimplementedanintermediatecycle(40r1.1)ofitsIntegratedForecastingSystem(IFS)thataddedthecapabilitytoactivelyassimilateallconventionalobservationaldatainBUFRformat(binarycode).ThismodificationwasneededsinceWMOallowedproviderstostopdisseminatingdatainTraditionalAlphanumericCodes(TAC)formatinNovember2014.WMOdecidedtomovetoarepresentationofobservationsinBUFRbecauseofthelackofflexibilityoftheTACformatusedfortheexchangeofsurfaceandupperairobservationsforthelast50years.ThenewBUFRformatallowssubstantiallymoredatatobeprovided,forexamplemuchhigherverticalresolutioninradiosondereports,thanwaspossiblewiththeTACformat.However,asignificantandcontinuingefforthasbeennecessarytomonitorthetransitiontothenewformatasnumerouserrorsandqualityissueshavebeenidentifiedasdataprovidersintroducethechange.ECMWFhasbeenplayinganactiveroleinthiseffortinclosecollaborationwiththeMemberStates,EUMETNET’sObservationsProgrammeandtheWMO.
Anewmodelcycle(41r1)wasimplementedon12May2015.
EvaluationofECMWFforecastsincluding2014‐2015upgrades
2 TechnicalMemorandumNo.765
Cycle41r1introducesalakeparametrization,basedontheFLakemodel,whichisappliedtoallresolvedandsub‐gridscalelakes.Theworkonlakesresultinginthisimplementationhasbenefittedfromamulti‐yearcollaborationwiththelake‐NWPcommunityinEuropeandinparticularrecognizesthescientificco‐ordinationoftheDeutscheWetterdienst.Thisimplementationimproves2‐metretemperatureforecastsinthevicinityofsmalllakesandnearcoastlinesnotrepresentedinthepreviousmodel.
Oceanwaveforecastsbenefitfromtheextensionofthehigh‐resolutionwavemodelfromtheEuropeanandNorthAtlanticregiontothewholeoftheglobe.Thesestand‐aloneforecastsaredrivenbythehighresolutionforecastHRES,areperformedatahigherresolutionthanthecoupledwavemodel,andincludeaforcingbyoceancurrents.
Newland‐seamask,orographyandclimatefields(glacierinformation,surfacealbedo)havebeenintroduced,aswellasnewdataforlakedepthandotherlakeparameters.ThenewmodelalsousesnewCO2,O3andCH4climatologiesfromthelatestMACC‐IIreanalysis.
Arevisedverticalinterpolationinthesemi‐Lagrangianadvectionschemereducesgravitywavenoiseduringsuddenstratosphericwarmingevents.
Theinner‐loopresolutionsofthe4DVARdataassimilationsystemhavebeenupgradedtoT255(80km)foreachofthethreeiterationsoftheouterloopstoproducefinerscaleincrements.Thebackgrounderrorcovariancesaremademoreflow‐dependentbyreducingthesamplingwindowandaveragingthestatisticsovershorterpastperiods,thesedynamicalstatisticsbeingusedjointlywithaclimatology.Arangeofadditionalsatelliteobservationsimprovestherepresentationoflandsurface,seaiceandoceanwaveparameters.
Monthlyensembleforecastsandre‐forecastshavebeenextendedfrom32to46days.Theextendedforecastsshouldbeusedwithcarebutresultshaveshownthatthereispositiveskillinsomeaspectsofforecastsinthe30–46dayrange.Theensembleforecast(ENS)re‐forecastdatasetissignificantlyenhanced,withre‐forecastsrunningtwiceaweek,forMondaysandThursdays(previouslyjustThursdays),andwiththesizeofeachre‐forecastensembleincreasedfrom5to11members.Thisprovidesasubstantialincreaseinthesamplesizeforthemodelclimatesforthemedium‐rangeExtremeForecastIndex(EFI)/ShiftofTails(SOT)andtheextended‐range(monthly)forecastanomalyproducts.
AsummaryofforecastperformanceisprovidedasascorecardinFigure1.
Thenewmodelcycleimprovesbothhigh‐resolutionforecasts(HRES)andensembleforecasts(ENS)throughoutthetroposphereandinthelowerstratosphere.Improvementsareseenbothinverificationagainstthemodelanalysisandverificationagainstobservations.
Cycle41r1bringsconsistentgainsinforecastperformanceatthesurfacefortotalcloudcoverandprecipitation.Improvementsinthemodellingofcloudandprecipitationreducethepredictedoccurrenceofdrizzleinsituationswherelarge‐scaleprecipitationdominates,andtheyincreasetheamountofrainfallinforecastsofintenseevents,leadingtoabettermatchwithobservations.Improvementsarealsoseenfor2‐metretemperatureand2‐metrehumidityinpartsofthenorthernhemisphereandthetropics.Cycle41r1alsointroducesanumberofnewoutputparameters,suchasprecipitationtype,includingfreezingrain.
Theaveragepositionerrorfortropicalcyclonesisslightlyreduced,andtropicalcyclonesaregenerallyforecasttobemoreintense.Forexample,IFSCycle41r1performedbetterthanCycle40r1inpredictingthetrackoftropicalcyclonePam,whichdevastatedVanuatuintheSouth
EvaluationofECMWFforecastsincluding2014‐2015upgrades
TechnicalMemorandumNo.765 3
PacificinMarch2015.InHRES,thesealevelpressureminimumatthecentreoftropicalcyclonesisonaverageslightlyloweratallleadtimes.Uptoandincludingday3thismakestheforecastbetter,byreducingtheslightpositivebias.Fromday5onwards,however,thepre‐existingbiastowardsover‐deepeninghasincreasedslightly.
Thenewmodelcycleisdescribedingreaterdetailat
http://www.ecmwf.int/en/forecasts/documentation‐and‐support/changesecmwf‐model/cycle‐41r1.
3. Verificationforupper‐airmedium‐rangeforecasts
3.1. ECMWFscoresFigure2showstheevolutionoftheskillofthehigh‐resolutionforecastof500hPaheightoverEuropeandtheextratropicalnorthernandsouthernhemispheressince1981.Eachpointonthecurvesshowstheforecastrangeatwhichthemonthlymean(bluelines)or12‐monthmeancentredonthatmonth(redline)oftheanomalycorrelation(ACC)betweenforecastandverifyinganalysisfallsbelow80%.InbothhemispheresandoverEuropescoreshavebeenconsistentlyhigh.Resulting12‐monthmeansareequaltoorslightlyexceedingthehighestpreviousvalues.
Acomplementarymeasureofperformanceistherootmeansquare(RMS)erroroftheforecast.Figure3showsRMSerrorsforbothextratropicalhemispheresofthesix‐dayforecastandthepersistenceforecast.Theerrorofthesix‐dayforecasthasfurtherdecreasedinthehemisphericaverages.
Figure4showsthetimeseriesoftheaverageRMSdifferencebetweenfour‐andthree‐day(blue)andsix‐andfive‐day(red)forecastsfromconsecutivedaysof500hPaforecastsoverEuropeandthenorthernextratropics.Thisillustratestheconsistencybetweensuccessive12UTCforecastsforthesameverificationtime;thegeneraldownwardtrendindicatesthatthereisless“jumpiness”intheforecastfromdaytoday.Thelevelofconsistencybetweenconsecutiveforecastshasincreasedfurtherinthelastyear.In2014the12‐monthmovingaveragesofRMSdifferencesreachedtheirlowestvaluessofar.
ThequalityofECMWFforecastsfortheupperatmosphereinthenorthernhemisphereextratropicsisshownthroughtimeseriesoftemperatureandwindscoresat50hPainFigure5.Scoresforone‐dayforecastsoftemperatureaswellasforecastsofvectorwindhavebeenstablesincelastyear.
Theverificationofmodelforecastsinthestratosphereiscurrentlyperformedagainstanalysesandradiosondeobservations.Bothdatasetshavelimitationsthatreduceourabilitytoproperlyassessmodeldevelopmentsinthestratosphere.GPSradiooccultation(RO)observationsrepresentanalternativewayofverificationinthestratosphere.Theyhaveagoodverticalresolution,globalandhomogeneousdistribution(around3000profilesperday),andtheydonotrequirebiascorrection.
SinceROmeasurementsareassimilateddirectlyintheformofbendingangles,oneoptionistoperformtheverificationusingthisquantity,whichisprimarilysensitivetovariationsintemperatureinthestratosphere.RObendinganglescanprovidearobustmeasureofforecasterrorchangesbetweenmodelcycles.Figure6(topleftpanel)showsthereductioninerrorstandarddeviationofbendinganglesduetothemostrecentmodelupgrade.Thebottompanel
EvaluationofECMWFforecastsincluding2014‐2015upgrades
4 TechnicalMemorandumNo.765
showsthecorrespondingtime‐seriesofbendingangledeparturesforbothmodelversions,indicatingthatthelargestreductionsintherandompartoftheforecasterroroccurduringstratosphericwarmingevents.
Sinceverificationresultsforbendinganglescanbedifficulttointerpret,temperatureretrievalsfromGPS‐ROrepresentanalternativewayofassessingtheimpactofmodelchangesonsystematictemperatureforecasterrorsinthelowerandmiddlestratosphere.However,GPS‐ROtemperatureretrievalsrequireaprioriinformationabouttheupperatmosphere.InordertohaverobustresultswhencomparingIFScyclesitisimportanttousethesamepriorinformationforROtemperatureretrievals,whichinthiscaseisprovidedbytheoperational6‐hourforecast.TheupperrightpanelinFigure6showsthereductioninthestandarddeviationofthetemperaturedeparturescorrespondingtotheimprovementinbendingangledeparturesshownintheleftpanel.
ThetrendinENSperformanceisillustratedinFigure7,whichshowstheevolutionofthecontinuousrankedprobabilityskillscore(CRPSS)for850hPatemperatureoverEuropeandthenorthernhemisphere.AsforHRES,theENSskillreachedrecordlevelsinwinter2009–10.Therehasbeensomereductionfromtheserecordlevels,especiallyoverEurope,asmightbeexpectedandaswasseenalsoforHRES.However,theENSperformancehasbeenconsistentlyhigh,andtheskillinwinter2013–14inthenorthernextratropicshasbeenverysimilartotherecordlevelsof2010.Anumberofchangeshavebeenmadetotheensembleconfigurationsince2010,includingimprovementstoboththeinitialperturbationsandrepresentationofmodeluncertainties,theincreaseinresolutioninJanuary2010,andfurtherredefinitionofperturbationsusingtheensembleofdataassimilations.Theslightlydecreasingtrendin2014isduetoatmosphericvariability.
Inawell‐tunedensemblesystem,theRMSerroroftheensemblemeanforecastshould,onaverage,matchtheensemblestandarddeviation(spread).Theensemblespreadandensemble‐meanerrorovertheextratropicalnorthernhemisphereforlastwinter,aswellasthedifferencebetweenensemblespreadandensemble‐meanerrorforthelastthreewinters,areshowninFigure8.Thematchbetweenspreadanderrorin2014issimilartopreviousyears,althoughslightlystrongerunder‐dispersioncanbeseeninthemediumrange.Theunder‐dispersionfortemperatureat850hPainbothseasonsisstillpresent,althoughuncertaintyintheverifyinganalysisshouldbetakenintoaccountwhenconsideringtherelationshipbetweenspreadanderrorinthefirstfewdays.
Agoodmatchbetweenspatiallyandtemporallyaveragedspreadanderrorisanecessarybutnotasufficientrequirementforawell‐calibratedensemble.Itshouldalsobeabletocaptureday‐to‐daychanges,aswellasgeographicalvariations,inpredictability.Thiscanbeassessedusingspread‐reliabilitydiagrams.Forecastvaluesofspreadoveragivenregionandtimeperiodarebinnedintoequallypopulatedspreadcategories,andforeachbintheaverageerrorisdetermined.Inawell‐calibratedensembletheresultinglineshouldbeclosetothediagonal.Figure9andFigure10showspread‐reliabilityplotsfor500hPageopotentialand850hPatemperatureinthenorthernextratropics(top),Europe(centre),andthetropics(bottom,inFigure10only)fordifferentglobalmodels.Spreadreliabilitygenerallyimproveswithleadtime.Atday1(leftpanels),forecaststendtobemorestronglyunder‐dispersiveatlowspreadvaluesthanatday6(rightpanels).ECMWFperformsverywell,withitsspreadreliabilityusuallyclosesttothediagonal.Thestarsintheplotsmarktheaveragevalues,correspondingtoFigure
EvaluationofECMWFforecastsincluding2014‐2015upgrades
TechnicalMemorandumNo.765 5
8,andideallyshouldlieonthediagonal,ascloselyaspossibletothelowerleftcorner.Alsointhisrespect,ECMWFperformsbestoverall.
InordertohaveabenchmarkfortheENS,theCRPShasbeencomputedfora‘dressed’ERA‐I.ThisalsohelpstodistinguishtheeffectsofIFSdevelopmentsfrompureatmosphericvariability.Thedressingusesthemeanerrorandstandarddeviationoftheprevious30daystogenerateaGaussiandistributionaroundtheERA‐I.Figure11showstheevolutionoftheCRPSfortheENSandforthedressedERA‐Ioverthelast10yearsfortemperatureat850hPaatforecastday5.InthenorthernhemispheretheskilloftheENSrelativetothereferenceforecastwasabout15%in2005andisapproaching30%in2015.ItisworthnotingthatusingtheforecasterrorfordressingoftheERA‐Iisequivalenttogeneratinganearlyperfectlycalibratedensemble.Thusthissortofreferenceforecastrepresentsachallengingbenchmark.
Theforecastperformanceoverthetropics,asmeasuredbyRMSvectorerrorsofthewindforecastwithrespecttotheanalysis,isshowninFigure12.At200hPa(upperpanel)the1‐dayforecasthascontinuedtoimproveslightly(althoughitisstillslightlyhigherthantheminimumwhichwasreachedin2003–2004),whilethe5‐dayforecasterrorhasincreased.Similarly,at850hPa(lowerpanel)theerroratday1hasbeenslightlyreducedwhileatday5ithasincreased.Theincreaseat850hPaisalsoseeninERA‐Interim(notshown)andinforecastsofothercentres.Itoccursforverificationagainstanalysisanddoesnotappearwhentheforecastisverifiedagainstobservations(cf.Section3.2,Figure16andFigure17).Notethatscoresforwindspeedinthetropicsaregenerallysensitivetointer‐annualvariationsoftropicalcirculationsystemssuchastheMadden‐Julianoscillation,orthenumberoftropicalcyclones.
3.2. WMOscores‐comparisonwithothercentresThecommongroundforcomparisonistheregularexchangeofscoresbetweenWMOdesignatedglobaldata‐processingandforecastingsystem(GDPFS)centresunderWMOcommissionforbasicsystems(CBS)auspices,followingagreedstandardsofverification.Thenewscoringproceduresforupper‐airfieldsusedintherestofthisreportwereapprovedforuseinthisscoreexchangebythe16thWMOCongressin2011andarenowbeingimplementedatparticipatingcentres.ECMWFceasedcomputationofscoresusingpreviousproceduresinDecember2011.ThereforetheECMWFscoresshowninthissectionareacombinationofscoresusingtheold(untilDecember2011)andnewprocedures(from2012onward).Thescoresfromothercentresfortheperiodofthisreporthavebeencomputedstillusingthepreviousprocedures.ForthescorespresentedheretheimpactofthechangesisrelativelysmallfortheECMWFforecastsanddoesnotaffecttheinterpretationoftheresults.
Figure13showstimeseriesofsuchscoresfor500hPageopotentialheightinthenorthernandsouthernhemisphereextratropics.Overthelast10yearserrorshavedecreasedforallmodels,especiallyduringthewinterseason.ECMWFcontinuestomaintainaleadovertheothercentres.
WMO‐exchangedscoresalsoincludeverificationagainstradiosondesoverregionssuchasEurope.Figure14(Europe),andFigure15(northernhemisphereextratropics)showingboth500hPageopotentialheightand850hPawindforecasterrorsaveragedoverthepast12months,confirmsthegoodperformanceoftheECMWFforecastsusingthisalternativereferencerelativetotheothercentres.
ThecomparisonforthetropicsissummarisedinFigure16(verificationagainstanalyses)andFigure17(verificationagainstobservations).Whenverifiedagainstthecentres’ownanalyses,
EvaluationofECMWFforecastsincluding2014‐2015upgrades
6 TechnicalMemorandumNo.765
theJapanMeteorologicalAgency(JMA)forecasthasthelowesterrorintheshortrange(day1)whileinthemediumrange,ECMWFandJMAaretheleadingmodelsinthetropics.Atthebeginningof2012theerrorsoftheECMWFforecastat850hPahaveshiftedtoaslightlylowerlevelduetoachangeinthecomputationofthescore.Insteadofsamplingthefullfieldsona2.5°grid,fieldsarenowspectrallytruncatedequivalentto1.5°resolution,inaccordancewithWMOguidelines.Inthetropics,verificationagainstanalyses(Figure16)isverysensitivetotheanalysis,inparticularitsabilitytoextrapolateinformationawayfromobservationlocations.Whenverifiedagainstobservations(Figure17),theECMWFforecasthasnowthesmallestoverallerrorsbothintheshortandmediumranges.
4. Weatherparametersandoceanwaves
4.1. Weatherparameters–high‐resolutionandensembleThesupplementaryheadlinescoresfordeterministicandprobabilisticprecipitationforecastsareshowninFigure18.Thetoppanelshowstheleadtimeatwhichthestableequitableerrorinprobabilityspace(SEEPS)skillforthehigh‐resolutionforecastforprecipitationaccumulatedover24hoursovertheextratropicsdropsbelow45%.Thisthresholdhasbeenchosensuchthatthescoremeasurestheskillataleadtimeof3–4days.ThebottompanelshowstheleadtimeatwhichtheCRPSSfortheprobabilityforecastofprecipitationaccumulatedover24hoursovertheextratropicsdropsbelow10%.Thisthresholdhasbeenchosensuchthatthescoremeasurestheskillataleadtimeofapproximately6days.Bothscoresareverifiedagainststationobservations.
MuchoftherecentvariationofthescoreforHRESisduetoatmosphericvariability,asshownbycomparisonwiththeERA‐Interimreferenceforecast(dashedlineinFigure18,toppanel).BytakingthedifferencebetweentheoperationalandERA‐Interimscoresmostofthisvariabilityisremoved,andtheeffectofmodelupgradesisseenmoreclearly(centrepanelinFigure18).Whilethelargestimprovementisassociatedwiththeintroductionofthefive‐speciesmicrophysicsinNovember2011(cycle36r4),microphysicschangesinsubsequentcyclesledtoafurtherincreaseinskill.Theprobabilisticscore(lowerpanelinFigure18)showssomerecentimprovementafterthestagnantperiod2010–2012whichwasagainpartlyduetoatmosphericvariability.TheCRPSoftheclimatologyforecast,whichisusedasareferencefortheCRPSS(seeAppendixA.2),decreased(i.e.improved)overtheperiod2010–2011,whichhasmaskedimprovementsduetomodelupgradesduringthattime.In2012,however,thistrendhasreversed,sothatmodelimprovementshavebecomemorevisibleagainintheCRPSS.
ECMWFperformsaroutinecomparisonoftheprecipitationforecastskillofECMWFandothercentresforboththehigh‐resolutionandtheensembleforecastsusingtheTIGGEdataarchivedintheMeteorologicalArchivalandRetrievalSystem(MARS).Resultsusingthesesameheadlinescoresforthelast12monthsshowtheHRESleadingwithrespecttotheothercentresfromday3onwardswhiletheMet‐Officemodelisleadingatday1(Figure19,upperpanel),andfortheENSaconsistentclearleadforECMWFoverthewholeleadtimerange(Figure19,bottompanel).
Trendsinmeanerrorandstandarddeviationoverthelast10yearsoferrorfor2mtemperature,2mdewpoint,totalcloudcover,and10mwindspeedforecastsoverEuropeareshowninFigure20toFigure23.VerificationisagainstsynopticobservationsavailableontheGlobalTelecommunicationSystem(GTS).Acorrectionforthedifferencebetweenmodel
EvaluationofECMWFforecastsincluding2014‐2015upgrades
TechnicalMemorandumNo.765 7
orographyandstationheightwasappliedtothetemperatureforecasts,butnootherpost‐processinghasbeenappliedtothemodeloutput.
Ingeneral,theperformanceoverthepastyearfollowsthetrendofpreviousyears.For2mtemperatureanddewpoint,theerrorstandarddeviation(uppercurvesineachplot)hasbeencomparativelysmall.However,negativebiaseswithmarkedannualcyclespersist,especiallyatnight‐time.Fortotalcloudcover(Figure22)thebiashasbeensmallinrecentyears,andtheerrorstandarddeviationhasshownlittlechange.Forwindspeed(Figure23)thereducedlevelofnight‐timebiasassociatedwiththechangeinsurfaceroughnessinNovember2011hasbeenmaintained.Howeverthedaytimebiashasbecomeslightlymorenegative.
Tocomplementtheevaluationofsurfaceweatherforecastskill,resultsobtainedforverificationagainstthetopoftheatmosphere(TOA)reflectedsolarradiationproducts(dailytotals)fromtheClimateMonitoringSatelliteApplicationFacility(CM‐SAF)basedonMeteosatdataareshown.Thereisanincreaseintheskilloftheoperationalhigh‐resolutionforecastrelativetoERA‐Interiminrecentyears,bothintheextratropicsandtropics(Figure24),thatcanbeattributedtothecombinedeffectofaseriesofmodelchangesbeginningwiththeintroductionofthefive‐speciesprognosticmicrophysicsschemeinNovember2010(cycle36r4).
ERA‐InterimisusefulasareferenceforecastfortheHRESasitallowsfilteringoutmuchoftheeffectofatmosphericvariationsonscores.Figure25showstheevolutionofskillatday5relativetoERA‐Interiminthenorthernhemisphereextratropicsforvariousupper‐airandsurfaceparameters.ThemetricusedistheRMSEforupper‐airfieldsandtheerrorstandarddeviationforthesurfacefields.Curvesshow12‐monthrunningmeanvalues.Itcanbeseenthatthelargestrelativeimprovements(15–20%since2002)havebeenachievedforupper‐airanddynamicfields,followedby2mtemperatureand10mwindspeed.Theskilloftotalcloudcover,whichhadbeenstablepriorto2011,startedtoincreaseasaresultofmorerecentcyclechanges.Formeansealevelpressure,thehighest12‐monthskillsofarwasreachedattheendof2014.Bothfor500hpageopotentialand850hPatemperature,maximasofaroccurredearlyin2014,andfurtherincreasesareexpectedfromthemodelupgradeinMay2015.
4.2. OceanwavesThequalityoftheoceanwavemodelanalysisandforecastisshowninthecomparisonwithindependentoceanbuoyobservationsinFigure26.ThetoppanelofFigure26showstimeseriesoftheforecasterrorfor10mwindspeedusingthewindobservationsfromthesebuoys.Theforecasterrorhassteadilydecreasedsince2001andithasreacheditslowestvaluesofarinthewinterseason2013–14.Errorsinthewaveheightforecastin2014–15havebeenthelowestsofarinthe1–3dayrange,andamongthelowestat5days.Thelong‐termtrendintheperformanceofthewavemodelforecastsisalsoseenintheverificationagainstanalysis.Anomalycorrelationforsignificantwaveheighthasreachedsomeofitshighestvaluesin2014(Figure27).
ECMWFmaintainsaregularinter‐comparisonofperformancebetweenwavemodelsfromdifferentcentresonbehalfoftheExpertTeamonWavesandStormSurgesoftheWMO‐IOCJointTechnicalCommissionforOceanographyandMarineMeteorology(JCOMM).Thevariousforecastcentrescontributetothiscomparisonbyprovidingtheirforecastsatthelocationsoftheagreedsubsetofoceanbuoys(mainlylocatedinthenorthernhemisphere).AnexampleofthiscomparisonisshowninFigure28forthe12‐monthperiodJune2013–May2014.ECMWFforecastwindsareusedtodrivethewavemodelofMétéoFrance;thewavemodelsofthetwo
EvaluationofECMWFforecastsincluding2014‐2015upgrades
8 TechnicalMemorandumNo.765
centresaresimilar,hencetheclosenessoftheirerrorsinFigure28.ECMWFoutperformstheothercentreswithregardtowindspeedandpeakperiod.Forwaveheight,MétéoFranceandECMWFforecastshavehighestskill.
AcomprehensivesetofwaveverificationchartsisavailableontheECMWFwebsiteat
http://www.ecmwf.int/en/forecasts/charts
under‘Oceanwaves’.
5. SevereweatherSupplementaryheadlinescoresforsevereweatherare:
TheskilloftheExtremeForecastIndex(EFI)for10mwindspeedverifiedusingtherelativeoperatingcharacteristicarea(Section5.1)
Thetropicalcyclonepositionerrorforthehigh‐resolutionforecast(Section5.2)
5.1. ExtremeForecastIndex(EFI)TheExtremeForecastIndex(EFI)wasdevelopedatECMWFasatooltoprovideearlywarningsforpotentiallyextremeevents.Bycomparingtheensembledistributionofachosenweatherparametertothemodel’sclimatologicaldistribution,theEFIindicatesoccasionswhenthereisanincreasedriskofanextremeeventoccurring.VerificationoftheEFIhasbeenperformedusingsynopticobservationsoverEuropefromtheGTS.Anextremeeventisjudgedtohaveoccurrediftheobservationexceedsthe95thpercentileoftheobservedclimateforthatstation(calculatedfroma15‐yearsample,1993–2007).TheabilityoftheEFItodetectextremeeventsisassessedusingtherelativeoperatingcharacteristic(ROC).Theheadlinemeasure,skilloftheEFIfor10mwindspeedatforecastday4(24‐hourperiod72–96hoursahead),isshowninFigure29(top),togetherwiththecorrespondingresultsfor24‐hourtotalprecipitation(centre)and2mtemperature(bottom).Eachcurveshowsseasonalvalues,aswellasthefour‐seasonrunningmean,ofROCareaskillscoresfrom2004to2014;thefinalpointoneachcurveincludesthespring(March–May)season2015.Forallthreeparameters,ROCskillhasstabilizedonahighlevel,withsomeinter‐annualvariationsduetoatmosphericvariability.
5.2. TropicalcyclonesThetropicalcyclonepositionerrorforthe3‐dayhigh‐resolutionforecastisoneofthetwosupplementaryheadlinescoresforsevereweather.Theaveragepositionerrorsforthehigh‐resolutionmedium‐rangeforecastsofalltropicalcyclones(alloceanbasins)overthelastten12‐monthperiodsareshowninFigure30.Errorsintheforecastintensityoftropicalcyclones,representedbythereportedsea‐levelpressureatthecentreofthesystem,arealsoshown.ThecomparisonofHRESandENScontroldemonstratesthebenefitofhigherresolutionfortropicalcycloneforecasts.
TheHRESandENSpositionerrors(topandbottompanels,Figure30)havereachedtheirlowestvaluessofar.ThesameistrueforthemeanabsolutespeederrorsoftheHRESandtheCTRLatD+3.Typicallytropicalcyclonesmovetooslowlyintheforecast,howeverthisnegativebiashasbeenrelativelysmallinrecentyears.Becauseofthesubstantialyear‐to‐yearvariationsinthenumberandintensityofcyclones,thereissomeuncertaintyinthesefigures.Boththemeanerror(bias)andmeanabsoluteerrorintropicalcycloneintensity(uppercentralpanelsin
EvaluationofECMWFforecastsincluding2014‐2015upgrades
TechnicalMemorandumNo.765 9
Figure30)haveincreased.Aswiththespeederrors,thereisarelativelylargeuncertaintyinthesescoresbecauseoftheyear‐to‐yearvariationsinthenumberandcharacterofstorms.
ThebottompanelofFigure30showsthespreadanderrorofensembleforecastsoftropicalcycloneposition.Forreference,theHRESerrorisalsoshown.Theforecastwasunder‐dispersivebeforetheresolutionupgradein2010,butthespread‐errorrelationshiphasimprovedsincethen.ThefigurealsoshowsthattheHRESpositionerrorhasbeengenerallysmallerthantheensemblemeanerroratforecastday3(althoughverysimilarrecently),andviceversaatforecastday5.
TheensembletropicalcycloneforecastispresentedontheECMWFwebsiteasastrikeprobability:theprobabilityatanylocationthatareportedtropicalcyclonewillpasswithin120kmduringthenext120hours.Verificationoftheseprobabilisticforecastsforthethreelatest12‐monthperiodsisshowninFigure31.Resultsshowacertainamountofover‐confidence,withlittlechangefromyeartoyear.SkillisshownbytheROCandthemodifiedROC,thelatterusingthefalsealarmratio(fractionofyesforecaststhatturnouttobewrong)insteadofthefalsealarmrate(ratiooffalsealarmstothetotalnumberofnon‐events)onthehorizontalaxis.Thisremovesthereferencetonon‐eventsinthesampleandshowsmoreclearlythereductioninfalsealarmsinthosecaseswheretheeventisforecast.Differencesbetweenthelastthreeconsecutiveyearsofthesetwomeasuresarenotconsideredsignificant.
5.3. Additionalsevere‐weatherdiagnosticsWhilemanyscorestendtodegeneratetotrivialvaluesforrareevents,somehavebeenspecificallydesignedtoaddressthisissue.Hereweusethesymmetricextremaldependenceindex,SEDI(AnnexA.4),toevaluateheavyprecipitationforecastskilloftheHRES.Forecastsareverifiedagainstsynopticobservations.Figure32showsthetime‐evolutionofskillexpressedintermsofforecastdaysfor24‐hourprecipitationexceeding20mminEurope.Thegaininskillamountstoabouttwoforecastdaysoverthelast15yearsandisprimarilyduetoahigherhitrate.AmoredetailedevaluationofheavyprecipitationforecastskillcanbefoundinECMWFNewsletterNo.144.
6. Monthlyandseasonalforecasts
6.1. MonthlyforecastverificationstatisticsandperformanceThemonthlyforecastingsystemhasbeenintegratedwiththemedium‐rangeensemblesinceMarch2008.Thecombinedsystemmadeitpossibletoprovideuserswithensembleoutputuniformlyupto32daysahead,onceaweek.AsecondweeklyrunofthemonthlyforecastwasintroducedinOctober2011,runningeveryMonday(00UTC)toprovideanupdatetothemainThursdayrun.InIFScycle41r1(May2015)themonthlyensembleforecastsandre‐forecastshavebeenextendedfrom32to46days.
Figure33showstheprobabilisticperformanceofthemonthlyforecastovertheextratropicalnorthernhemisphereforsummer(JJA,toppanels)andwinter(DJF,bottompanels)seasonssinceSeptember2004forweek2(days12–18,leftpanels)andweek3+4(days19–32rightpanels).CurvesshowtheROCscorefortheprobabilitythatthe2mtemperatureisintheupperthirdoftheclimatedistributioninsummer,andinthelowerthirdoftheclimatedistributioninwinter.Thusitisameasureoftheabilityofthemodeltopredictwarmanomaliesinsummerandcoldanomaliesinwinter.Forreference,theROCscoreofthepersistenceforecastisalsoshownineachplot.Forecastskillforweek2exceedsthatofpersistencebyabout10%,for
EvaluationofECMWFforecastsincluding2014‐2015upgrades
10 TechnicalMemorandumNo.765
weeks3to4(combined)byabout5%.Intheweeks3to4(14‐dayperiod),summerwarmanomaliesappeartohaveslightlyhigherpredictabilitythanwintercoldanomalies,althoughthelatterhasincreasedinrecentwinters(withtheexceptionof2012).Overall,theskilloftheforecastismorestablefromyeartoyearthantheskillofpersistence,bothinsummerandwinter.
ComprehensiveverificationforthemonthlyforecastsisavailableontheECMWFwebsiteat:
http://www.ecmwf.int/en/forecasts/charts
6.2. Seasonalforecastperformance
6.2.1. SeasonalforecastperformancefortheglobaldomainThecurrentversion(System4)oftheseasonalcomponentoftheIFSwasimplementedinNovember2011.ItusestheoceanmodelNEMOandECMWFatmosphericmodelcycle36r4.Theforecastscontain51ensemblemembersandthere‐forecasts15ensemblemembers,coveringaperiodof30years.
Asetofverificationstatisticsbasedonre‐forecastintegrations(1981–2010)fromSystem4hasbeenproducedandispresentedalongsidetheforecastproductsontheECMWFwebsite.
AcomprehensivedescriptionandassessmentofSystem4isprovidedinECMWFTechnicalMemorandum656,availablefromtheECMWFwebsite.
6.2.2. The2014–2015ElNiñoforecastsTheyear2014wascharacterizedbyachangefromslightlycoldtoslightlywarmconditionsintheeasterntropicalPacific.Themajorityofensemblemembersoftheforecastsmadeinspringandsummerof2014(uppertwopanelsintheleftcolumnofFigure34)predictedamoresubstantialstrengtheningofwarmanomaliesthanwhatwasobserved.Theautumnforecastcapturedthebasiccharacteristicsofthenextmonths’evolutionbetter,suggestinglittlechange.Thewinterforecast(bottomleftpanel)againpredictedastrongevolutiontowardsElNinoconditions,whichthistimeagreedverywellwithobservations.Themulti‐modelEUROSIPforecasts(rightcolumnofFigure34)performedslightlybetterinthefirsthalfoftheperiod,inthesensethattheensemblewasbettercentredontheobservations.TheFeb2015forecastofthestrongElNinodevelopmentin2015waspredictedmoreclearly(withgreatersharpness)intheECMWFmodel.
TheECMWFforecastsystempredictsthecontinuedstrengtheningoftheElNinointhecomingmonths,withthepeakexpectedaroundDecember2015.PreviousexperienceshowsthatforverylargeElNinoevents(specifically1997)themodeltendstoexaggeratetheamplitudeofSSTanomaliesduetonon‐linearitiesinthemodelbiascharacteristics.The2015ElNinoislikelytobeverystrong,butmorelikelythannotstillweakerthantheonein1997.Thelongerrangeforecast(13monthslead)suggeststhatElNinoislikelytoendsometimeinApril/May/June2016,withtemperatureseithernormalorbelownormalbyJuly2016.
6.2.3. TropicalstormpredictionsfromtheseasonalforecastsThe2014NorthAtlantichurricaneseasonwasquietwithanaccumulatedcycloneenergyindex(ACE)ofjust67%ofthe1950–2012climateaverage(seeFigure35).Thenumberoftropicalstormswhichformedin2014(8namedstorms)wasalsobelowaverage(12).SeasonaltropicalstormpredictionsfromSystem4indicatedbelowaverageactivitycomparedtoclimatologyover
EvaluationofECMWFforecastsincluding2014‐2015upgrades
TechnicalMemorandumNo.765 11
theAtlantic.TheJuneforecastpredicted9(witharangefrom6to12)tropicalstormsintheAtlantic(Figure36)andanACEof60%oftheobservedclimatology(+/‐20%).Mostotherseasonalforecastmodelspredictedabelowaverage2014AtlantictropicalstormseasonduetothepresenceofamoderateEl‐Ninoevent.
Figure36showsthatSystem4predictedaboveaverageactivityovertheeasternNorthPacific(althoughwiththeACE10%belownormal)andslightlyenhancedactivityoverthewesternNorthPacific(ACE20%abovenormal).The2014easternPacifichurricaneseasonwaswellaboveaverage(19tropicalstormsformedovertheeasternNorthPacificbetweenJulyandDecember)andtheACEwas43%abovethe1981–2010average,whichmakesittheseventh‐highestsince1971.Only17tropicalstormsformedoverthewesternNorthPacificin2014betweenJulyandDecember,whichisbelowaverage(21.3).Therewasnoclearsignalintheforecastoverthisbasin.ThedropoftheACEintheAtlanticsectorwaswellcapturedbytheforecast,withalmostthesameamplitudeasobserved,althoughmostensemblememberspredictedatthetimeastrongerEl‐Nino(conducivetoareductionintropicalcycloneactivityintheAtlantic)thanobserved(Figure34).
6.2.4. Extratropicalseasonalforecasts2mtemperaturesinthenorthern‐hemispherewinter(DJF2014–15)werecharacterizedbystrongwarmanomaliesovernorthernEurasia,whichwerecapturedquitewellbytheseasonalforecast(Figure37).ThewesternpartsofEurope,incontrast,wereinfluencedbyapersistentcoldanomalyovertheNorthAtlantic.ThisanomalywasalsocapturedbythemodelbutitdidnotextendquiteasfarintoEuropeaswasobserved.Theseasonalforecastwasnotabletopredictthelarge‐scalecoldanomalyovermuchoftheeasternUnitedStatesandCanada.
Duringthenorthern‐hemispheresummer(JJA2015),centralandsouthernEuropeexperiencedextremelypersistentheat,leadingtoamagnitudeofseasonal2mtemperatureanomalyofmorethan2standarddeviationsabovenormal(Figure38).InsomeEuropeancountriesall‐timerecordtemperaturesweresurpassed.TemperaturesinnorthernEuropewereatthesametimebelownormal,withasteepgradientoftheanomalyatabout50‐55degreesnorth.TheseasonalforecastpredictedpositiveanomaliesinsouthernEurope,andnon‐significantanomaliesinnorthernEurope.TheextensionoftheanomalouswarmthintoAsiawaswellcaptured.
EvaluationofECMWFforecastsincluding2014‐2015upgrades
12 TechnicalMemorandumNo.765
anomalycorrelation RMS error SEEPS
Europe
against analysis
Geopotential
100hPa ▴▲▴▴▴▴░░░░ ▲▲▲▲▲▴▴░░░ 500hPa ▴▴▴░░░░░░░ ▲▴▴▴▴░░░░░ 850hPa ▴▴▴░░░░░░░ ▴▲▴▴▴░░░░░ 1000hPa ▴▴▴░▴▴░░░░ ▲▲▴▴▴░░░░░
MSL pressure ░▴▴░░░░░░░ ▴▴▴░░░░░░░
Temperature
100hPa ▲▲▲▴▴▴░░░░ ▲▲▲▲▲▲▴▴░▴ 500hPa ▴▴▴▴░░░░░░ ▴▴▴▴▴▴░░░░ 850hPa ▲▴▴░▴▴░░░░ ▲▴▴░▴░░░░░ 1000hPa ▴▴▴▴▴▴░░░░ ▴▴▴▴▴▴▴░░░
Wind 200hPa ▲▴▴▴▴▴░░░░ ▲▲▴▴▴░░░░░ 850hPa ▴▴▴▴▴░░░░░ ▴▴▴▴▴░░░░░
Relative humidity
300hPa ▴░▴░░░░░░░ ▴▴▴░░░░░░░ 700hPa ▲▴▴▴▴▴░░░░ ▲▲▲▴▴▴░░▴░
against observations
Temperature
100hPa ░░░░░░░░░░ ▲▲▲▲▴▴▴░░░ 200hPa ░▴▴░░░░░░░ ░░▴░░░░░░░ 850hPa ▴░░░░░░░░░ ░░░░░░░░░░
2m temperature ▴▴▲▴▴▴▴░▴░
Wind
100hPa ▴▴▴░░░░░░░ ▴▴░░░░░░░░ 200hPa ░▴░░░░░░░░ ░░░░░░░░░░ 850hPa ░░░░░░░░░░ ░▴░░░░░░░░
10m wind
░░░░░░░░░░ 2m dew-point ▾░░▴▴░▴▴▴░ Total cloud cover ▲▲▲▴░▴▴░░░ 24h precipitation ▴▴▴░░░░░░░
Extratropical Northern Hemisphere
against analysis
Geopotential
100hPa ▲▲▲▲▲▴▴▴▴▴ ▲▲▲▲▲▲▲▲▲▴ 500hPa ▲▲▲▴▴▴▴░░░ ▲▲▲▴▴▴▴▴▴░ 850hPa ▲▲▲▴▴▴▴▴░░ ▲▲▲▴▴▴▴▴▴▴ 1000hPa ▲▲▴▴▴▴▴▴░░ ▲▲▴▴▴▴▴▴░░
MSL pressure ▴▴▴░░▴▴░░░ ▴▴▴░░▴▴░░░
Temperature
100hPa ▲▲▲▲▴▴▴▴▴▴ ▲▲▲▲▲▲▲▲▲▲ 500hPa ▲▲▲▴▴▴▴▴░░ ▲▲▲▴▴▴▴▴▴▴ 850hPa ▲▴▴▴▴▴▴░░░ ▴░░░░░░░░░ 1000hPa ▴▴▴▲▴▴▴▴▴░ ▲▲▲▲▴▴▴▴▴▴
Wind 200hPa ▲▲▲▴▴▴▴▴▴▴ ▲▲▲▲▴▴▴▴▴▴ 850hPa ▲▴▴▴▴▴▴▴▴▴ ▲▴▴░▴▴▴▴▴▴
10m wind over ocean
▴▴░░░░▴░▴▴ ░░░░░░░░▴▴
Ocean wave height ▼▾▾░░▴▴░░░ ▼▾░░░░▴░▴░
Ocean wave period ▼▼▼▼▾░░░░░ ▼▼▼▼▾▾░░░░
Relative humidity
300hPa ▲▴▴▴▴▴░░░░ ▲▴▴▴▴▴▴▴▴▴ 700hPa ▲▴▴▴▴▴░░▴▴ ▲▲▲▲▲▲▲▲▲▲
against observations
Temperature
100hPa ▴▴▴▴░░░░░░ ▲▲▲▲▲▲▲▴▴▴ 200hPa ▴▴▴▴░░░░░░ ▴▴▴░░░░░░░ 850hPa ▾▾░░░░░░░░ ▾▾░░░░░░░░
2m temperature ▲▲▲▲▴▴▴▴▴░
Wind
100hPa ▲▲▴▴▴░░░░░ ▲▲▴▴▴▴░░░░ 200hPa ▲▴▴▴░░░░░░ ▲▴▴▴░░░░░░ 850hPa ▾░░░░░░░░░ ▾░░░░░░░░░
10m wind
▲▲▲▲▴▴░░░░ 2m dew-point ▴▲▲▲▲▲▴▴▴▴ Total cloud cover ▴▴▴▴▴▴▴▴▴▴ 24h precipitation ░░░░░░░░░░
Extratropical Southern Hemisphere
against analysis
Geopotential
100hPa ▴▴▴▴▴▴▴▴░░ ▲▲▲▲▲▲▲▲▲▲ 500hPa ▲▲▲▴▴▴▴▴▴▴ ▲▲▲▴▴▴▴▴▴▴ 850hPa ▲▲▲▴▴▴▴▴▴▴ ▲▲▲▴▴▴▴▴▴▴ 1000hPa ▲▲▲▴▴▴▴▴▴▴ ▲▲▲▴▴▴▴▴▴▴
MSL pressure ▲▲▴▴▴▴▴▴▴▴ ▲▲▴▴▴▴▴▴▴▴
Temperature
100hPa ▲▲▲▲▲▲▴▴▴▴ ▲▲▲▲▲▲▲▲▲▲ 500hPa ▲▲▲▴▴▴▴░▴▴ ▲▲▲▲▴▴▴▴▴▴ 850hPa ▲▴▴░░░░░░░ ░▾▾▾▾▾░▾▾▾ 1000hPa ▲▲▲▲▴▴▴▴▴░ ▲▲▴▴▴░░░░░
EvaluationofECMWFforecastsincluding2014‐2015upgrades
TechnicalMemorandumNo.765 13
anomalycorrelation RMS error SEEPS
Wind 200hPa ▴▴▴▴▴▴▴▴░░ ▲▴▴▴▴▴▴▴▴▴ 850hPa ▲▲▴▴▴▴▴▴▴▴ ▲▲▲▴▴▴▴▴▴▴
10m wind over ocean
▲▴▴▴▴▴▴▴▴▴ ▴▴▴▴▴░░░░▴
Ocean wave height ▼▼░░░░░▴▴▴ ▼▼░░░░░▴░▴
Ocean wave period ▼▼▼▼▼▾░░░░ ▼▼▼▼▼▾▾▾░░
Relative humidity
300hPa ▴▴▴▴▴▴▴▴▴▴ ░░▴▴▴▴▴▴▴▴ 700hPa ▲▴▴▴▴░░░░░ ▲▲▲▲▲▲▲▲▲▲
against observations
Temperature
100hPa ▴▴▴▴▴▴▴▴▴▴ ▲▲▲▲▲▲▲▲▲▲ 200hPa ▴▴▴▴▴▴▴▴▴░ ▴▴▴▴▴░▴▴▴░ 850hPa ░░░░░░▴░░░ ░░░░░░▴░░░
2m temperature ▴▴▴▴▴▴▴▴░░
Wind
100hPa ▲▲▲▴▴▴▴░░░ ▲▲▲▲▴▲▴▴▴▴ 200hPa ░▴▴▴▴▴▴▴░░ ░▴▴▴▴▴▴▴░▴ 850hPa ▴▴░▴░▴░░░░ ▴▴▴▴▴▴▴░░░
10m wind
▾▼▾▾▾░░░░░ 2m dew-point ░░▾░░░▴░░░ Total cloud cover ▲▴▴▴▴▴▴░▴░ 24h precipitation ░░░░░░▴░░░
Tropics
against analysis
Temperature
100hPa ▲▲▲▲▲▲▲▲▲▲ ▲▲▲▲▲▲▲▲▲▲
500hPa ▲▲▲▲▲▲▲▲▲▲ ▲▲▲▲▲▲▲▲▲▲
850hPa ▲▲▴▴▴▴▴░░░ ▲▴░░░░░░░░ 1000hPa ░▾▾░░░░░▾▾ ▲▲▲▲▲▲▲▲▲▴
Wind 200hPa ▲▲▲▲▲▲▲▲▲▲ ▲▲▲▲▲▲▲▲▲▲
850hPa ▲▲▲▲▲▲▲▴▴▴ ▲▲▲▴▴▴▴▴░░ 10m wind over ocean
▲▲▲▲▲▲▲▴▴▴ ▲▲▲▲▴▴░░▾░
Ocean wave height ▼▼▼▼▼▾▾▾▾▾ ▼▼▼▼▼▼▼▼▼▾
Ocean wave period ▼▼▼▼▼▾▾▾▾░ ▼▼▼▼▼▼▼▼▾▾
Relative humidity
300hPa ░▴▴▴▴▴▴▴▴▴ ▼▼▾▾▾▾▾▾▾▾ 700hPa ▲▴▴▴▴▴▴░░░ ▲▲▲▲▲▲▲▲▲▲
against observations
Temperature
100hPa ▲▲▲▲▲▲▲▲▲▴ ▲▲▲▲▲▲▲▲▲▲ 200hPa ▲▲▲▲▲▲▲▴▲▴ ▲▴░░░░░░░░ 850hPa ░░░░░░░░░░ ▴▴▴▴▴▴▴▴░░
2m temperature ▲▲▲▲▲▲▲▲▲▲
Wind
100hPa ▲▲▲▲▲▲▲▴▴▴ ▲▲▲▲▲▲▲▲▲▲ 200hPa ▴░▴▴▴▴▴░░░ ▴░▴▴▴▴▴▴▴▴ 850hPa ▲▲▲▴▴▴▴▴▴▴ ▴▴▴▴▴▴░░░░
10m wind
▼▼▼▼▼▼▼▼▼▾ 2m dew-point ▲▲▲▲▲▲▲▲▲▲
Total cloud cover ▲▲▲▲▲▲▲▲▲▲
24h precipitation ▴▴▴▴░░░░░░ Symbol legend: for a given forecast step... (d: score difference, s: confidence interval width) ▲ CY41r1 better than CY40r1 statistically highly significant (the confidence bar above zero by more than its height) (d/s>3) ▴ CY41r1 better than CY40r1 statistically significant (d/s≥1) ░ CY41r1 better than CY40r1, yet not statistically significant (d/s≥0.5) ░ not really any difference between CY40r1 and CY41r1 ░ CY41r1 worse than CY40r1, yet not statistically significant (d/s≤-0.5) ▾ CY41r1 worse than CY40r1 statistically significant (d/s≤-1) ▼ CY41r1 worse than CY40r1 statistically highly significant (the confidence bar below zero by more than its height) (d/s<-3)
Figure1: SummaryscorecardforCy41r1.Scorecardforcycle41r1versuscycle40r1verifiedbytherespectiveanalysesandobservationsat00and12UTCfor704forecastrunsintheperiod2January2014to10May2015.
EvaluationofECMWFforecastsincluding2014‐2015upgrades
14 TechnicalMemorandumNo.765
Figure2:Primaryheadlinescore for thehigh‐resolution forecasts.Evolutionwith timeof the500hPageopotentialheight forecastperformance–eachpointon thecurves is the forecast rangeatwhich themonthlymean(bluelines)or12‐monthmeancentredonthatmonth(redline)oftheforecastanomalycorrelation (ACC) with the verifying analysis falls below 80% for Europe (top), northern hemisphereextratropics(centre)andsouthernhemisphereextratropics(bottom).
Europe
NH
SH
EvaluationofECMWFforecastsincluding2014‐2015upgrades
TechnicalMemorandumNo.765 15
Figure3:Rootmeansquare(RMS)errorofforecastsof500hPageopotentialheight(m)atday6(red),verifiedagainstanalysis.Forcomparison,areferenceforecastmadebypersistingtheanalysisover6daysisshown(blue).Plottedvaluesare12‐monthmovingaverages;thelastpointonthecurvesisforthe12‐month periodAugust 2014–July 2015. Results are shown for the northern extra‐tropics (top), and thesouthernextra‐tropics(bottom).
NH
SH
EvaluationofECMWFforecastsincluding2014‐2015upgrades
16 TechnicalMemorandumNo.765
Figure 4: Consistency of the 500 hPa height forecasts over Europe (top) and northern extratropics(bottom).CurvesshowthemonthlyaverageRMSdifferencebetweenforecastsforthesameverificationtimebutinitialised24hapart,for96–120h(blue)and120–144h(red).12‐monthmovingaveragescoresarealsoshown(inbold).
Europe
NH
EvaluationofECMWFforecastsincluding2014‐2015upgrades
TechnicalMemorandumNo.765 17
Figure 5: Model scores for temperature (top) and wind (bottom) in the northern extratropicalstratosphere.CurvesshowthemonthlyaverageRMStemperatureandvectorwinderrorat50hPaforone‐day(blue)andfive‐day(red)forecasts,verifiedagainstanalysis.12‐monthmovingaveragescoresarealsoshown(inbold).
50hPatemperature
50hPawindspeed
EvaluationofECMWFforecastsincluding2014‐2015upgrades
18 TechnicalMemorandumNo.765
Figure6:Normaliseddifferencesbetweenmodelcycles41r1and40r1ofthestandarddeviationofday5forecastdepartures(normalisedbytheobservationerror)ofGPSRObendingangles(upperleftpanel)andGPSROtemperatureretrievals(upperrightpanel)overthenorthernhemisphereextra‐tropics.Valuesbelow100indicateareductioninthestandarddeviationcomparedtothereference.Thebottompanelshowscorrespondingtimeseriesofthestandarddeviationof5‐dayforecastdeparturesofGPSRObendinganglesforthelayerbetween40and49km(blue:40r1,red:41r1).
EvaluationofECMWFforecastsincluding2014‐2015upgrades
TechnicalMemorandumNo.765 19
Figure7:Primaryheadlinescorefortheensembleprobabilisticforecasts.Evolutionwithtimeof850hPatemperature ensemble forecast performance, verified against analysis. Each point on the curves is theforecastrangeatwhichthe3‐monthmean(bluelines)or12‐monthmeancentredonthatmonth(redline)of the continuous ranked probability skill score (CPRSS) falls below 25% for Europe (top), northernhemisphereextratropics(bottom).
EvaluationofECMWFforecastsincluding2014‐2015upgrades
20 TechnicalMemorandumNo.765
Figure8:Ensemble spread (standarddeviation,dashed lines)andRMSerrorofensemble‐mean (solidlines) forwinter2014–2015(upper figure ineachpanel),anddifferencesofensemblespreadandRMSerrorofensemblemeanforlastthreewinterseasons(lowerfigureineachpanel,negativevaluesindicatespreadistoosmall);verificationisagainstanalysis,plotsarefor500hPageopotential(top)and850hPatemperature(bottom)overtheextratropicalnorthernhemisphereforforecastdays1to15.
EvaluationofECMWFforecastsincluding2014‐2015upgrades
TechnicalMemorandumNo.765 21
Figure9:Ensemblespreadreliabilityofdifferentglobalmodelsfor500hPageopotentialinDJF2014–15inthenorthernhemisphereextra‐tropics(top)andinEurope(bottom)forday1(left)andday6(right),verifiedagainstanalysis.Circlesshowerrorfordifferentvaluesofspread,starsshowaverageerror‐spreadrelationship.
EvaluationofECMWFforecastsincluding2014‐2015upgrades
22 TechnicalMemorandumNo.765
Figure10:Ensemblespreadreliabilityofdifferentglobalmodelsfor850hPatemperatureinDJF2014–15inthenorthernhemisphereextra‐tropics(top),Europe(centre),andthetropics(bottom)forday1(left)andday6(right),verifiedagainstanalysis.Circlesshowerrorfordifferentvaluesofspread,starsshowaverageerror‐spreadrelationship.
EvaluationofECMWFforecastsincluding2014‐2015upgrades
TechnicalMemorandumNo.765 23
Figure11:CRPSfortemperatureat850hPainthenorthern(top)andsouthern(bottom)extratropicsatday5,verifiedagainstanalysis.Scoresareshownfortheensembleforecast(red)andthedressedERA‐Interimforecast(blue).BlackcurvesshowtheskilloftheENSrelativetothedressedERA‐Interimforecast.Valuesarerunning12‐monthaverages.NotethatforCRPS(redandbluecurves)lowervaluesarebetter,whileforCRPSskill(blackcurve)highervaluesarebetter.
EvaluationofECMWFforecastsincluding2014‐2015upgrades
24 TechnicalMemorandumNo.765
Figure12:Forecastperformanceinthetropics.CurvesshowthemonthlyaverageRMSvectorwinderrorsat200hPa(top)and850hPa(bottom)forone‐day(blue)andfive‐day(red) forecasts,verifiedagainstanalysis.12‐monthmovingaveragescoresarealsoshown(inbold).
200hPa
850hPa
EvaluationofECMWFforecastsincluding2014‐2015upgrades
TechnicalMemorandumNo.765 25
Figure13:WMO‐exchangedscoresfromglobalforecastcentres.RMSerrorof500hPageopotentialheightovernorthern(top)andsouthern(bottom)extratropics.Ineachpaneltheuppercurvesshowthesix‐dayforecasterrorandthelowercurvesshowthetwo‐dayforecasterror.Eachmodelisverifiedagainstitsownanalysis.JMA=JapanMeteorologicalAgency,CMC=CanadianMeteorologicalCentre,UKMO=theUKMetOffice,NCEP=U.S.NationalCentersforEnvironmentalPrediction,M‐F=MétéoFrance.
EvaluationofECMWFforecastsincluding2014‐2015upgrades
26 TechnicalMemorandumNo.765
Figure14:WMO‐exchangedscoresforverificationagainstradiosondes:500hPaheight(top)and850hPawind(bottom)RMSerroroverEurope(annualmeanAugust2014–July2015).
EvaluationofECMWFforecastsincluding2014‐2015upgrades
TechnicalMemorandumNo.765 27
Figure15:AsFigure14forthenorthernhemisphereextratropics.
EvaluationofECMWFforecastsincluding2014‐2015upgrades
28 TechnicalMemorandumNo.765
Figure16:WMO‐exchangedscores fromglobal forecastcentres.RMSvectorwinderrorovertropicsat250hPa(top)and850hPa(bottom).Ineachpaneltheuppercurvesshowthefive‐dayforecasterrorandthelowercurvesshowtheone‐dayforecasterror.Eachmodelisverifiedagainstitsownanalysis.
EvaluationofECMWFforecastsincluding2014‐2015upgrades
TechnicalMemorandumNo.765 29
Figure17:AsFigure16forverificationagainstradiosondeobservations.
EvaluationofECMWFforecastsincluding2014‐2015upgrades
30 TechnicalMemorandumNo.765
Figure18:Supplementaryheadlinescoresfordeterministic(top,centre)andprobabilistic(bottom)precipitationforecasts.Theevaluationisfor24‐hourtotalprecipitationverifiedagainstsynopticobservationsintheextratropics;eachpointiscalculatedovera12‐monthperiod,plottedatthecentreoftheperiod.ThedashedcurveshowsthedeterministicheadlinescoreforERA‐Interimasareference.ThecentrepanelshowsthedifferencebetweentheoperationalforecastandERA‐Interim.Curvesshowthenumberofdaysforwhichthecentred12‐monthmeanskillremainsaboveaspecifiedthreshold.Theforecastdayonthey‐axisistheendofthe24‐hourperiodoverwhichtheprecipitationisaccumulated.
SEEPS
ERA‐Interim
SEEPSdifference
HRES
HRES‐ERA‐Interim
CRPSS
EvaluationofECMWFforecastsincluding2014‐2015upgrades
TechnicalMemorandumNo.765 31
Figure19:ComparisonofprecipitationforecastskillforECMWF(red),theMetOffice(UKMO,blue),JapanMeteorologicalAgency (JMA,magenta) andNCEP (green) using the supplementaryheadline scores forprecipitation shown in Figure 18. Top: deterministic; bottom: probabilistic skill. Curves show the skillcomputedoverallavailablesynopticstationsintheextratropicsforforecastsfromAugust2014–July2015.Barsindicate95%confidenceintervals.
HRESprecipitation‐SEEPS
ENSprecipitation‐CRPSS
EvaluationofECMWFforecastsincluding2014‐2015upgrades
32 TechnicalMemorandumNo.765
Figure20:Verificationof2mtemperatureforecastsagainstEuropeanSYNOPdataontheGTSfor60‐hour(night‐time)and72‐hour(daytime)forecasts.Lowerpairofcurvesshowsbias,uppercurvesarestandarddeviationoferror.
Figure 21: Verification of 2m dewpoint forecasts against European SYNOP data on the GlobalTelecommunicationSystem(GTS)for60‐hour(night‐time)and72‐hour(daytime)forecasts.Lowerpairofcurvesshowsbias,uppercurvesshowstandarddeviationoferror.
2‐mtemperature
2‐mdewpoint
EvaluationofECMWFforecastsincluding2014‐2015upgrades
TechnicalMemorandumNo.765 33
Figure22:VerificationoftotalcloudcoverforecastsagainstEuropeanSYNOPdataontheGTSfor60‐hour(night‐time) and 72‐hour (daytime) forecasts. Lower pair of curves shows bias, upper curves showstandarddeviationoferror.
Figure23:Verificationof10mwindspeedforecastsagainstEuropeanSYNOPdataontheGTSfor60‐hour(night‐time) and 72‐hour (daytime) forecasts. Lower pair of curves shows bias, upper curves showstandarddeviationoferror.
Totalcloudcover
10‐mwindspeed
EvaluationofECMWFforecastsincluding2014‐2015upgrades
34 TechnicalMemorandumNo.765
Figure24:12‐monthrunningaverageoftheday3forecastskillrelativetoERA‐InterimofnormalizedTOAreflectedsolarflux(dailytotals),verifiedagainstsatellitedata.Theverificationhasbeencarriedoutforthose parts of the northern hemisphere extratropics (green), tropics (red), and southern hemisphereextratropics(blue)whicharecoveredbytheCM‐SAFproduct(approximately70Sto70N,and70Wto70E).
EvaluationofECMWFforecastsincluding2014‐2015upgrades
TechnicalMemorandumNo.765 35
Figure25:EvolutionofskilloftheHRESforecastatday5,expressedasrelativeskillcomparedtoERA‐Interim.Verificationisagainstanalysisfor500hPageopotential(Z500),850hPatemperature(T850),andmeansealevelpressure(MSLP),usingRMSEasametric.VerificationisagainstSYNOPfor2mtemperature(T2M),10mwindspeed(V10),andtotalcloudcover(TCC),usingerrorstandarddeviationasametric.
EvaluationofECMWFforecastsincluding2014‐2015upgrades
36 TechnicalMemorandumNo.765
Figure26: Time series of verification of the ECMWF10mwind forecast (top panel) andwavemodelforecast(waveheight,bottompanel)verifiedagainstnorthernhemispherebuoyobservations.Thescatterindexistheerrorstandarddeviationnormalisedbythemeanobservedvalue;athree‐monthrunningmeanisused.
EvaluationofECMWFforecastsincluding2014‐2015upgrades
TechnicalMemorandumNo.765 37
Figure27:Oceanwaveforecasts.Monthlyscoreand12‐monthrunningmean(bold)ofACCforoceanwaveheightsverifiedagainstanalysisforthenorthern(top)andsouthernextratropics(bottom)atday1(blue),5(red)and10(green).
EvaluationofECMWFforecastsincluding2014‐2015upgrades
38 TechnicalMemorandumNo.765
Figure28:Verificationofdifferentmodelforecastsofwaveheight,10mwindspeedandpeakwaveperiodusingaconsistentsetofobservationsfromwavebuoys.Thescatterindex(SI)isthestandarddeviationoferrornormalisedbythemeanobservedvalue;plotsshowtheSIforthe12‐monthperiodJuly2014–June2015.Thex‐axisshowstheforecastrangeindaysfromanalysis(step0)today5.MOF:MetOffice,UK;FNM:FleetNumericalMeteorologyandOceanographyCentre,USA;NCP:NationalCenters forEnvironmentalPrediction, USA; MTF: Météo‐France; DWD: Deutscher Wetterdienst, BoM: Bureau of Meteorology,Australia;JMA:JapanMeteorologicalAgency;KMA:KoreaMeteorologicalAdministration.
EvaluationofECMWFforecastsincluding2014‐2015upgrades
TechnicalMemorandumNo.765 39
Figure 29: Verification of Extreme Forecast Index (EFI) against analysis. Top panel: supplementaryheadlinescore–skilloftheEFIfor10mwindspeedatforecastday4(24‐hourperiod72–96hoursahead);anextremeevent is takenasanobservationexceeding95thpercentileof stationclimate.Curvesshowseasonalvalues(dotted)andfour‐seasonrunningmean(continuous)ofrelativeoperatingcharacteristic(ROC) area skill scores. Centre and bottom panels show the equivalent ROC area skill scores forprecipitationEFIforecastsandfor2mtemperatureEFIforecasts.
2‐mtemperature
10‐mwindspeed
24‐hprecipitation
EvaluationofECMWFforecastsincluding2014‐2015upgrades
40 TechnicalMemorandumNo.765
Figure30:Verificationoftropicalcyclonepredictionsfromtheoperationalhigh‐resolutionandensembleforecast. Results are shown for all tropical cyclones occurring globally in 12‐month periods ending on30June. Verification is against the observed position reported via the GTS. Top panel supplementaryheadlinescore–themeanpositionerror(km)ofthethree‐dayhigh‐resolutionforecast.Theerrorforday5isincludedforcomparison.Centrefourpanelsshowmeanerror(bias)inthecycloneintensity(differencebetweenforecastandreportedcentralpressure;positiveerrorindicatestheforecastpressureislessdeepthanobserved),meanabsoluteerroroftheintensityandmeanandabsoluteerrorofcyclonemotionspeedforcycloneforecastbothbyHRESandENScontrol.Bottompanelshowsmeanpositionerrorofensemblemean(meanofcyclonesforecastbyensemblemembers)withrespecttotheobservedcyclone(cyancurve)andensemblespread(meanofdistancesofensemblecyclonesfromtheensemblemean;redcurve);forcomparisontheHRESpositionerror(fromthetoppanel)isplottedaswell(bluecurve).
EvaluationofECMWFforecastsincluding2014‐2015upgrades
TechnicalMemorandumNo.765 41
Figure31:Probabilisticverificationofensembletropicalcycloneforecastsatday10forthree12‐monthperiods:July2012–June2013(green),July2013–June2014(blue)andJuly2014–June2015(red).Upperpanelshowsreliabilitydiagram(theclosertothediagonal,thebetter).Thelowerpanelshows(left)thestandardROCdiagramand(right)amodifiedROCdiagram,wherethefalsealarmratioisusedinsteadofthefalsealarmrate.ForbothROCandmodifiedROC,thecloserthecurveistotheupper‐leftcorner,thebetter,indicatingagreaterproportionofhits,andfewerfalsealarms.
EvaluationofECMWFforecastsincluding2014‐2015upgrades
42 TechnicalMemorandumNo.765
Figure32:EvolutionofskilloftheHRESforecastinpredicting24‐hprecipitationamounts>20mmintheextra‐tropicsasmeasuredbytheSEDIscore,expressedintermsofforecastdays.VerificationisagainstSYNOPobservations.NumbersontherightindicatedifferentSEDIthresholdsused.Curvesshow12‐monthrunningaverages.
EvaluationofECMWFforecastsincluding2014‐2015upgrades
TechnicalMemorandumNo.765 43
Figure 33: Verification of the monthly forecast against analysis. Area under the ROC curve for theprobabilitythat2mtemperatureisintheupperthirdoftheclimatedistributioninsummer(top)andinthelowerthirdinwinter(bottom).Scoresarecalculatedforeachthree‐monthseasonforalllandpointsintheextra‐tropicalnorthernhemisphere.Leftpanelsshowthescoreoftheoperationalmonthlyforecastingsystemforforecastdays12–18(7‐daymean),andrightpanelsforforecastdays19–32(14‐daymean).Asa reference, lighter coloured lines shows the scoreusingpersistence of thepreceding7‐day or 14‐dayperiodoftheforecast.
EvaluationofECMWFforecastsincluding2014‐2015upgrades
44 TechnicalMemorandumNo.765
Figure34:ECMWF(leftcolumn)andEUROSIPmulti‐modelforecast(rightcolumn)seasonalforecastsofSSTanomaliesovertheNINO3.4regionofthetropicalPacificfrom(toptobottomrows)May2014,August2014,November2014andFebruary2015.Theredlinesrepresenttheensemblemembers;dottedbluelineshowsthesubsequentverification.
EvaluationofECMWFforecastsincluding2014‐2015upgrades
TechnicalMemorandumNo.765 45
Figure35:Timeseriesofaccumulatedcycloneenergy(ACE)fortheAtlantictropicalstormseasonsJuly–December1990toJuly–December2014.Bluelineindicatestheensemblemeanforecastsandgreenbarsshowtheassociateduncertainty(±1standarddeviation);reddottedlineshowsobservations.ForecastsarefromSystem4oftheseasonalcomponentoftheIFS:thesearebasedonthe15‐memberre‐forecasts;from2011onwardstheyarefromtheoperational51‐memberseasonalforecastensemble.Startdateoftheforecastis1June.
EvaluationofECMWFforecastsincluding2014‐2015upgrades
46 TechnicalMemorandumNo.765
Figure36:TropicalstormfrequencyforecastissuedinJune2014forthesix‐monthperiodJuly–December2014.Greenbarsrepresenttheforecastnumberoftropicalstormsineachoceanbasin(ensemblemean);orangebarsrepresentclimatology.Thevaluesofeachbararewritteninblackunderneath.Theblackbarsrepresent±1standarddeviationwithintheensembledistribution;thesevaluesareindicatedbythebluenumber.The51‐memberensembleforecastiscomparedwiththeclimatology.AWilcoxon‐Mann‐Whitney(WMW)testisthenappliedtoevaluateifthepredictedtropicalstormfrequenciesaresignificantlydifferentfromtheclimatology.TheoceanbasinswheretheWMWtestdetectssignificancelargerthan90%haveashadedbackground.
EvaluationofECMWFforecastsincluding2014‐2015upgrades
TechnicalMemorandumNo.765 47
Figure37:Anomalyof2mtemperatureaspredictedbytheseasonalforecastfromNovember2014forDJF2014/15(upperpanel),andverifyinganalysis(lowerpanel).Blackcontoursintheanalysisindicateregionswhereanomaliesexceed1.5standarddeviations.
Evaluationof ECMWFforecastsincluding2014‐2015upgrades
48 TechnicalMemorandumNo.765
Figure38:Anomalyof2mtemperatureaspredictedbytheseasonalforecastfromMay2015forJJA2015(upperpanel),andverifyinganalysis(lowerpanel).Blackcontoursintheanalysisindicateregionswhereanomaliesexceed1.5standarddeviations.
EvaluationofECMWFforecastsincluding2014‐2015upgrades
TechnicalMemorandumNo.765 49
Ashortnoteonscoresusedinthisreport
A.1 Deterministicupper‐airforecastsTheverificationsusedfollowWMOCBSrecommendationsascloselyaspossible.Scoresarecomputedfromforecastsonastandard1.5×1.5grid(computedfromspectralfieldswithT120truncation)limitedtostandarddomains(boundingco‐ordinatesarereproducedinthefigureinnercaptions),asthisistheresolutionagreedintheupdatedWMOCBSrecommendationsapprovedbythe16thWMOCongressin2011.Whenothercentres’scoresareproduced,theyhavebeenprovidedaspartoftheWMOCBSexchangeofscoresamongGDPScentres,unlessstatedotherwise–e.g.whenverificationscoresarecomputedusingradiosondedata(Figure14),thesondeshavebeenselectedfollowinganagreementreachedbydatamonitoringcentresandpublishedintheWMOWWWOperationalNewsletter.
Rootmeansquareerrors(RMSE)arethesquarerootofthegeographicalaverageofthesquareddifferencesbetweentheforecastfieldandtheanalysisvalidforthesametime.Whenmodelsarecompared,eachmodelusesitsownanalysisforverification;RMSEforwinds(Figure14,Figure16)arecomputedbytakingtherootofthesumsofthemeansquarederrorsforthetwocomponentsofthewindindependently.
SkillscoresarecomputedasthereductioninRMSEachievedbythemodelwithrespecttopersistence(forecastobtainedbypersistingtheinitialanalysisovertheforecastrange);inmathematicalterms:
2
2
1*100p
f
RMSE
RMSESS
Figure2showscorrelationsinspacebetweentheforecastanomalyandtheverifyinganalysisanomaly.AnomalieswithrespecttoERA‐InterimanalysisclimateareavailableatECMWFfromearly1980s.Foroceanwaves(Figure27)theclimatehasbeenalsoderivedfromtheERA‐Interimanalyses.
A.2 ProbabilisticforecastsEventsfortheverificationofmedium‐rangeprobabilisticforecastsareusuallydefinedasanomalieswithreferencetoasuitableclimatology.Forupper‐airparameters,theclimateisderivedfromERA‐Interimanalysesforthe20‐yearperiod1989–2008.Probabilisticskillisevaluatedinthisreportusingthecontinuousrankedprobabilityskillscore(CRPSS)andtheareaunderrelativeoperatingcharacteristic(ROC)curve.
Thecontinuousrankedprobabilityscore(CRPS),anintegralmeasureofthequalityoftheforecastprobabilitydistribution,iscomputedas
where isforecastprobabilitycumulativedistributionfunction(CDF)and isanalysedvalue
expressedasaCDF.CRPSiscomputeddiscretelyfollowingHersbach,2000.CRPSSisthencomputedas
Evaluationof ECMWFforecastsincluding2014‐2015upgrades
50 TechnicalMemorandumNo.765
1
whereCRPSclimistheCRPSofaclimateforecast(basedeitherontheERA‐Interimanalysisorobservedclimatology).CRPSSisusedtomeasurethelong‐termevolutionofskilloftheIFSensemble(Figure7)anditsinter‐annualvariability(Figure11).
ROCcurvesshowhowmuchsignalcanbegainedfromtheensembleforecast.Althoughasinglevaluedforecastcanbecharacterisedbyauniquefalsealarm(x‐axis)andhitrate(y‐axis),ensembleforecastscanbeusedtodetectthesignalindifferentways,dependingonwhethertheforecastuserismoresensitivetothenumberofhits(theforecastwillbeissued,evenifarelativelysmallnumberofmembersforecasttheevent)oroffalsealarms(onewillthenwaitforalargeproportionofmemberstoforecasttheevent).TheROCcurvesimplyshowsthefalsealarmandhitratesassociatedwiththedifferentthresholds(proportionofmembersorprobabilities)used,beforetheforecastisissued(Figure31).Figure31alsoshowsamodifiedROCplotofhitrateagainstfalsealarmratio(fractionofyesforecaststhatturnouttobewrong)insteadofthefalsealarmrate(ratiooffalsealarmstothetotalnumberofnon‐events).
Sincetheclosertotheupperleftcorner(0falsealarm,100%hits)thebetter,theareaundertheROCcurve(ROCA)isagoodindicationoftheforecastskill(0.5isnoskill,1isperfectdetection).TimeseriesoftheROCAareshowninFigure33.
A. 3 Weather parameters (Section 4) VerificationofthedeterministicprecipitationforecastsismadeusingthenewlydevelopedSEEPSscore(Rodwelletal.,2010).SEEPS(stableequitableerrorinprobabilityspace)usesthreecategories:dry,lightprecipitation,andheavyprecipitation.Here“dry”isdefined,withreferencetoWMOguidelinesforobservationreporting,tobeanyaccumulation(roundedtothenearest0.1mm)thatislessthanorequalto0.2mm.Toensurethatthescoreisapplicableforanyclimaticregion,the“light”and“heavy”categoriesaredefinedbythelocalclimatologysothatlightprecipitationoccurstwiceasoftenasheavyprecipitation.Aglobal30‐yearclimatologyofSYNOPstationobservationsisused(theresultingthresholdbetweenthelightandheavycategoriesisgenerallybetween3and15mmforEurope,dependingonlocationandmonth).SEEPSisusedtocompare24‐houraccumulationsderivedfromglobalSYNOPobservations(exchangedovertheGlobalTelecommunicationSystem;GTS)withvaluesatthenearestmodelgrid‐point.1‐SEEPSisusedforpresentationalpurposes(Figure18,Figure19)asthisprovidesapositivelyorientedskillscore.
TheensembleprecipitationforecastsareevaluatedwiththeCRPSS(Figure18,Figure19).VerificationisagainstthesamesetofSYNOPobservationsasusedforthedeterministicforecast.
Forotherweatherparameters(Figure20toFigure23),verificationdataareEuropean6‐hourlySYNOPdata(areaboundariesarereportedaspartofthefigurecaptions).Modeldataareinterpolatedtostationlocationsusingbi‐linearinterpolationofthefourclosestgridpoints,providedthedifferencebetweenthemodelandtrueorographyislessthan500m.AcrudequalitycontrolisappliedtoSYNOPdata(maximumdeparturefromthemodelforecasthastobelessthan25K,20g/kgor15m/sfortemperature,specifichumidityandwindspeedrespectively).2mtemperaturesarecorrectedfordifferencesbetweenmodelandtrue
EvaluationofECMWFforecastsincluding2014‐2015upgrades
TechnicalMemorandumNo.765 51
orography,usingacrudeconstantlapserateassumptionprovidedthecorrectionislessthan4Kamplitude(dataareotherwiserejected).
A.4 VerificationofrareeventsExperimentalverificationofdeterministicforecastsofrareeventsisperformedusingthesymmetricextremaldependenceindexSEDI(Figure32),whichiscomputedas
log log log 1 log 1log log log 1 log 1
whereFisthefalsealarmrateandHisthehitrate.InordertoobtainafaircomparisonbetweentwoforecastingsystemsusingSEDI,theforecastsneedtobecalibrated(FerroandStephenson,2011).ThereforeSEDIisameasureofthepotentialskillofaforecastsystem.Inordertogetafullerpictureoftheactualskill,thefrequencybiasoftheuncalibratedforecastcanbeanalysed.
ReferencesFerro,C.A.T.,andD.B.Stephenson,2011:Extremaldependenceindices:improvedverificationmeasuresfordeterministicforecastsofrarebinaryevents.Wea.Forecasting,26,699–713.
Hersbach,H., 2000:Decomposition of the Continuous Ranked Probability Score for EnsemblePredictionSystem.Wea.Forecasting,15,559–570.
Richardson,D. S., 2000: Skill and relative economic value of the ECMWFensemble predictionsystem.Q.J.R.Meteorol.Soc.,126,649–667.
Rodwell,M.J.,D.S.Richardson,T.D.Hewson,andT.Haiden,2010:Anewequitablescoresuitableforverifyingprecipitationinnumericalweatherprediction.Q.J.R.Meteorol.Soc.,136,1344–1363.