Applications of the Golden Rule

J. Scott Armstrong*, University of Pennsylvania Kesten C. Green*, University of South Australia

*Ehrenberg-Bass Institute at University of South Australia

International Symposium on ForecastingRiverside, California

24 June 2015Slides available at ForPrin.com

A&G ISF 2015 – Golden-R11

Golden Rule of Forecasting: “Be Conservative”

or “Forecast unto others as you would have them

forecast unto you.”

Be conservative by adhering to cumulative knowledge about:1.the situation, and2.evidence-based forecasting methodsThe “Golden Rule of Forecasting” was published in June 2015.

2

Golden Rule of Forecasting (GR) Guidelines

Procedure: By logic, we developed 28 guidelines.

Validity testing by analyzing prior experimental comparisons relevant to the guidelines, almost none of which were done in awareness of the Golden Rule.

Directional effects were consistent with comparative tests of accuracy.

70 papers tested effect sizes: On average, the use of a single guideline reduced forecast error by 31%.

3

Why experimental findings?

We believe that experimentation is the basis for scientific advances.

Not feasible to identify causality from analyses of nonexperimental data in uncertain complex situations. Illusions in regression analysis

Directions of effects from nonexperimental studies often differ with those from experimental studies. Armstrong & Patnaik (2009).

4

Golden Rule based on cumulative knowledge about forecasting methods

Proved to be a large undertaking to develop the hypothesis and to accurately summarize the evidence. 1.Required three years to complete the paper.2.Eighteen people provided peer review.3.Eleven researchers contributed on various aspects.4.Four editors worked on the writing.

5

Support for the Golden Rule

1. We contacted all authors of key studies for whom we found email addresses.

2. Of those, 84% responded. 3. All but one agreed with our summary of

their work (an issue not to be taken for granted. See “Fawlty Towers” paper).

6

Example: Predicting Election outcomesThe Golden Rule Checklist was used to evaluate PollyVote. 1.Independent judgments were made as to whether the situation involved:

a) complexity Modest b) uncertainty Modest c) likelihood of bias Low

2.Independent judgments were made as to which guidelines were relevant, and3.Ratings were made as to whether the the Golden Rule was used properly or not.4.Comparisons were made of the accuracy of alternative methods.

7

Forecast Accuracy of the PollyVote vs. typical econometric model

We rated the PollyVote against the Golden Rule checklist.

13 of the 28 guidelines were relevant to forecasting elections.

The PollyVote adheres to all 13 guidelines.

8

Forecast Accuracy of the PollyVote vs. typical econometric model

(across remaining days to election, 1992-2012)

9

Climate Change Forecasts

We rated two methods used to forecast global mean temperatures.

10

The Chart Behind Global Warming:1981-2013(by Anthony Watts)

This is the surface temperature record, on the scale of human experience.

Warming alarmists do not forecast, they create “scenarios” via computer simulations1. Scenarios are:

a. Stories… about “what happened in the future”b. Biased… so do not provide valid forecasts

(Gregory & Duran, 2001).2. The stories are based on expert judgments. 3. According to prior research, expert judgments

about what will happen in complex, uncertain situations are no more accurate than forecasts from people with little expertise:a. Seer-sucker Theoryb. Tetlock’s 20-year experiment

12

No-change model is conservative given cumulative knowledge about the situation

Disagreement about the effects of the many variables that affect temperature.See e.g. NIPCC’s Climate Change Reconsidered II: Physical Science& Climate Change: The Facts 2014

13

Golden Rule applied to IPCC scenarioGolden Rule of Forecasting Checklist was used to evaluate IPCC “business as usual” global warming scenario and no-change model forecasts.

Consensus ratings by Armstrong and Green indicated that of the 20 relevant Golden Rule Checklist guidelines:• the IPCC scenarios followed none• the no-change model followed 95%

Don’t believe us? Rate them yourself and send us your ratings and reasons!

Tests of forecast accuracy over the 1851-1975 forecasting period yielded 58 forecasts for horizons of 91 to 100 years.

Average error (MAE) of no-change forecast for 50-year horizon was 0.24°C.

Errors from the IPCC scenario of .03°C warming-per-year were 12.6 times larger than those from the no-change model forecasts.

14

Example of a conservative guideline

1.1.2 Decompose to best use prior information. Look for data where there are different causal causal factors (e.g., to forecast traffic deaths forecast miles driven and deaths per mile driven, then recombine as shown in this paper.)

Is the weather getting warmer in Las Vegas?

Each day’s high and low temperatures are averaged. Than an average is taken across all days.

15

17

Does everyone agree that Las Vegas has gotten warmer since 1937?

Any complainers?

20

Why is it warmer at night in Las Vegas, but cooler in the day?

During the day, the pavements and buildings store the heat.

The heat is emitted during the night.

Our thanks to Anthony Watts for this example.

Earlier evidence on accuracy of IPCC projections vs. no-change forecasts

Tests of forecasts over the 1851-1975 forecasting period yielded 58 forecasts for horizons of 91 to 100 years.

The errors of these IPCC forecasts were 12.6 times larger than those from the simpler no-change model.

21

Conclusions for Climate Change

1. Alarming IPCC temperature projections are based on procedures that are insufficiently conservative to be trusted

2. Cumulative knowledge about the situation was ignored. In the belief that “this time it is different.”

22

Possible Applications

Golden Rule checklist allows commentators and decision makers to assess whether forecasters were forecasting as they would expect others to forecast unto them. Especially useful for areas subject to bias, such as

a. corporate mergers, b. mass transportation systems, c. law suits (advertising: Lance Armstrong case)d. public policies (e.g., gun control, minimum

wages).23

Conclusions on the use of the Golden Rule Checklist

1. Rapid application of evidence developed over the past century.

2. Inexpensive (raters don’t need high expertise in forecasting)

3. Use of the guidelines produces more accurate ex ante forecasts

4. Avoids the biggest forecast errors (e.g. Winter Storm) 5. Leads to actions that can improve accuracy.6. Allows commentators and decision makers to assess

whether forecasters were forecasting as they would expect others to forecast unto them.

24