Transition plan for MAPP/CTB Funded Proposal:Improved probabilistic forecast products for the NMME seasonal forecast system Anthony Barnston (PI), Huug van den Dool, Emily Becker, Michael Tippett, Shuhua Li DemonstrationThe project aims to improve the NMME probabilistic forecasts by addressing systematic biases in both forecast anomalies and categorical probabilities. The corrected NMME forecasts will have improved reliability and accuracy. The improvements will come about due to (1) corrections in pattern placement andpattern characteristics (developed at IRI), and (2) refinements to local probability anomalies (developed at CPC). The development work toward (1) and (2), currentlyin progress, will be done during year 1 and the first half of year 2.

Operational DeploymentDuring the last quarter of the last year, the software associated with the NMME corrections/improvements will be amalgamated at NOAA/CPC in order that they be implemented in real-time on a monthly basis there. Part of this software consolidation may require rewriting some codes in a different programming language, such as converting Matlab scripts to Fortran code.

The CPC component of the proposalProbabilistic forecast products for the NMME seasonal forecast system

Emily Becker and Huug van den Doolalso Li-Chuan Chen

THIS IS WHAT WE DO INREAL TIME

In the same way that traditional MSE can be minimized by a regression, we here attempt to minimize the BS, i.e. the MSE for probability forecasts.

The meaning/interpretation of traditional anomaly correlation (AC)By extension the meaning/interpretation of the probability anomaly correlation (PAC)

Tool to be applied Result1

Deterministic AC Forecast gets damped towards deterministic climatology=long term mean

Probabilistic PAC Forecast gets damped towards probabilistic climatology=(1/3rd,1/3rd,1/3rd)

Tool to be applied Result2

Deterministic AC Lower MSE

Probabilistic PAC Lower BS

Unanswered question: To what extent should we be ruled by verification metrics?

Brier Score A N Bunadj 0.122 0.154 0.132

adj 0.116 0.140 0.124

Brier Skill Sc. A N Bunadj 0.146 -0.068 0.115

adj 0.192 0.027 0.164

CFSv2 JanIC SST forecasts for February, Northern HemisphereHindcasts 1982-2010

.The PAC trick works : lower BS = higher accuracy

.It cleans up –ve skill in N class, no embarrassment.

.It tones down too bold forecasts, particularly in A&B class

.The gain is modest, but is the impact small?

BS = Reliability minus Resolution plus Uncertainty

Brier Score A N Bunadj 0.122 0.154 0.132

adj 0.116 0.140 0.124

Brier Skill Sc. A N Bunadj 0.146 -0.068 0.115

adj 0.192 0.027 0.164

CFSv2 JanIC SST forecasts for February, Northern Hemisphere

Brier Score A N Bunadj 0.122 0.154 0.132

adj 0.116 0.140 0.124

Brier Skill Sc. A N Bunadj 0.146 -0.068 0.115

adj 0.192 0.027 0.164

CFSv2 JanIC SST forecasts for February, Northern Hemisphere

BS = Reliability minus Resolution plus Uncertainty

In conclusion (CPC part)

• Probability Anomaly Correlation approach yields a lower BS (as expected) for any model and even for NMME collection. This can be implemented!

• PAC approach is being at CPC compared to ensemble regression, to have some reference.

• PAC approach has a large impact on the reliability-resolution diagram. Puzzling actually.

• PAC has an interesting application in verification of ENSO composites (Li-Chuan Chen)

• An adjustment to our most beloved conclusion

.How much improvement to be expected by (any) calibration?.Correctibility? Inherent skill>=threshold.Individual Models may improve more than the NMME collection.

Managing expectations

Quad Chart to Track Progress/Issues of MAPP-CTB Demonstration Project in FY16Improved probabilistic forecast products for the NMME seasonal forecast system

Issues/RisksIssues:1) **.

Mitigation:

Finances

SchedulingProject Information and Highlights

Funding Sources:FY14: $271KFY15: $91K

Status:

Management Attention Required

Potential Management Attention Needed

On Target

Updated on 11/06/2015

Lead:Columbia University: Anthony Barnston , Michael Tippett, Shuhua LiNCEP/CPC: Huug van den Dool, Emily Becker

Scope: Improve NMME-based probabilistic forecast products

Estimated Benefits:Improved NMME probabilistic forecasts by addressing systematic biases in both forecast anomalies and categorical probabilities. Software will be deployed at CPC

Milestone Date Status

Corrections in pattern placement and pattern characteristics

FY16Q1

Refinements to local probability anomalies (developed at CPC).

FY16Q3

Post-project review FY16Q4

GYR

G G

GY

(A. Barnston, M. Tippett, S. Li at IRI)

Use of multivariate statistical methods to correct systematic errors in placement of forecast patterns of each model.

Calibration steps: Spatial corrections by IRI may precede local probability adjustments by CPC.

Uncorrected model forecast Corrected model forecast

Example: Precipitation forecast during El Nino, DJF, from one atmospheric model

Disruption of progress at IRI in 2015

During year 1 of this CTB project, Mike Tippett took a 75%-time teaching position at Columbia University, leaving only summers to do research. Somuch of his funding was transferred to Shuhua Li. Mike has been trainingShuhua to use Matlab in the intended context of this project. Learninghas been steady but slow.

In the last month, Shuhua Li gave notice to leave IRI, taking an operationalforecasting job at BoM in Australia. So his training in this project will notbear any fruit.

Due to the above delay and then unexpected loss of Shuhua Li, IRI presentlyhas no concrete results to show regarding model spatial corrections.

Possible fixes for these setbacks: (1) Barnston will do all of the hands-onwork using CPT instead of Matlab (a quicker, but possibly less permanent, software package); (2) We will request a 1-year no-cost extension of the project, and find a new person to do the hands-on model correction trials.