point, interval, and density forecast evaluation of linear...

Point, Interval, and Density Forecast Evaluationof Linear versus Nonlinear DSGE Models

Francis X. Diebold Frank Schorfheide Minchul Shin

University of Pennsylvania

May 4, 2014

1 / 33

Motivation

The use of DSGE models as a forecasting tool in practice calls forevaluation of its performance.

Summary of DSGE forecasting evaluation literature:

I Point forecasts are as good as other statistical models such asVAR.

I Density forecasts are too wide.

Typically, DSGE models in this literature are

I Solved with linearized method.

I Constant volatility.

Forecasting with a nonlinear DSGE model:

I Pichler (2008) considers the second-order perturbationmethod.

2 / 33

Nonlinearity and Forecasting

Not much work has been done for nonlinear DSGE forecasting.

Two types of nonlinearitiesI Time-varying volatility vs. Constant volatility

I There are many papers on DSGE models with time-varyingvolatility but there is no paper that systematically evaluatesforecasts.

I Most papers report that density forecasts generated from theDSGE models with constant volatility is too wide.

I Nonlinear approximation vs. Linear approximationI Pichler (2008)’s point forecast evaluation.I Interval and density forecasts evaluation.

3 / 33

Summary

MethodI Small-scale DSGE model.

I Linear DSGE model with constant volatility.I Linear DSGE model with time-varying volatility.I Quadratic DSGE model (based on the second-order perturbation

method).I Quadratic DSGE model with time-varying volatility.

I Bayesian estimation and forecasting. (US data, 1964-2011)

Finding

I Modelling time-varying volatility improves accuracy offorecasts.

I Second-order perturbation method does not improve.

4 / 33

Model, Estimation, and Forecasting

Small-Scale DSGE Model

Small-scale model used in Herbst and Schorfheide (2012)

I Euler equation, NK Phillips curve, Monetary policy rule

I 3 exogenous shocks: technology, government spending,monetary policy (zt , gt ,mpt)

Measurement equations:(YGRt

INFt

FFRt

)= D(θ) + Z (θ) st

Transition equations:

st = Φ(st−1, εt ; θ)

where

st = [yt , yt−1, ct , πt ,Rt ,mpt , zt , gt ]′

εt : Innovations

θ : DSGE parameters

6 / 33

Linear DSGE Model

Linear approximation methods lead to a linear Gaussian state spacerepresentation with the following transition equation

st = H(θ)st−1 + R(θ)εt , εt ∼ iidN (0,Q(θ)).

I Coefficient matrices (H(θ),R(θ),Q(θ)) are the nonlinearfunction of θ.

I We obtain posterior draws based on the Random WalkMetropolis (RWM) algorithm with the Kalman filter.

7 / 33

Linear+Stochastic Volatility

Following Justiniano and Primiceri (2008),

st = H(θ)st−1 + R(θ)εt , εt ∼ N (0,Qt(θ))

where

diag(Qt(θ)) = [e2hmp,t , e2hz,t , e2hg,t ]′

hi ,t = ρσihi ,t−1 + νi ,t , νi ,t ∼ iidN (0, s2i ),

for i = mp, g , z .

I The system is in a linear Gaussian state-space formconditional on Qt(θ).

I We use the Metropolis-within-Gibbs algorithm developed byKim, Shephard, and Chib (1998) to generate draws from theposterior distribution.

8 / 33

Quadratic DSGE Model

The equilibrium law of motion based on the second-orderperturbation method has the following form:

s1,t = G0(θ) + G1(θ)s2,t + G2(θ)(s2,t ⊗ s2,t)

s2,t = H0(θ) + H1(θ)s2,t−1 + H2(θ)(s2,t−1 ⊗ s2,t−1) + R(θ)εt

where εt ∼ iidN (0,Q(θ)), st = [s1,t , s2,t ]′ and ⊗ is a Kronecker

product.

I We have an additional quadratic term.

I We need a nonlinear filter to get posterior draws.

I Particle filter + RWM (exact).I Second-order Extended Kalman Filter + RWM (approximate).

I Current version of the paper utilizes the second-order extended Kalmanfilter and RWM.

I We also consider Quadratic DSGE model with stochastic volatility.

9 / 33

Predictive Distribution

We generate draws from the posterior predictive distribution basedon the following decomposition,

p(YT+1:T+H |Y1:T ) =

∫p(YT+1:T+H |θ,Y1:T )p(θ|Y1:T )dθ

I p(θ|Y1:T ) : Posterior sampler.

I p(YT+1:T+H |θ,Y1:T ) : Given θ, simulate the model economyforward.

Draws {Y (j)T+1:T+H}

nsimj=1 can be turned into point and interval

forecasts by the Monte Carlo average,

E[yT+h|T

]=

∫yT+h

yT+h p(yT+h|Y1:T )dyT+h ≈1

nsim

nsim∑j=1

y(j)T+h.

10 / 33

Evaluation Methods

Point Forecast

Point forecasts from two different models (Inflation rate)

We compare RMSEs for variable i = {YGR, INFL, FFR},

RMSE(i |h) =

√√√√ 1

P − h

R+P−h∑t=R

(yi,t+h − yi,t+h|t)2

where R is the index denotes the starting point of the forecasts evaluationsample and P the number of forecasting origins.

12 / 33

Interval Forecast

Interval forecasts from two different models (Inflation rate)

I We compute α% interval forecasts for a particular element yi,T+h of yT+h

by numerically searching for the shortest connected interval that containsa α% of the draws {y (j)

i,T+h}nsimj=1 (the highest-density set).

I If the interval forecast is well calibrated, actual variables are expected tobe inside of α% interval forecasts at the same frequency.

13 / 33

Interval Forecast

Interval forecasts from two different models (Inflation rate)

Christoffersen (1998)’s LR tests of the correct coverage (α):

I Define the sequence of hit indicators of a 1-step-ahead forecast interval,

Iαt = 1{realized yt falls inside the interval}

I TestIαt ∼ iidBernoulli(α).

14 / 33

Density Forecast

Density forecasts from two different models (Inflation rate)

The probability integral transform (PIT) of yi,T+h based on time T predictivedistribution is defined as the cumulative density of the random variable,

zi,h,T =

∫ yi,T+h

−∞p(yi,T+h|Y1:T )dyi,T+h.

I If the predictive distribution is well-calibrated, zi,h,T should follow theuniform distribution.

I For h = 1, zi,h,T ’s follow independent uniformly distribution.

I Diebold, Gunther, and Tay (1998).

15 / 33

Density Forecast Evaluation: Predictive Likelihood

The one-step-ahead predictive likelihood,

PL(t) = p(yt+1|Y1:t)

I Height of the predictive density at the realized value yt+1.

I Log predictive score:∑R+P−1

t=R logPL(t).

We approximate

PL(t) ≈ 1

M

M∑m=1

p(yt+1|Y1:t , θ

(m))

where {θ}(m) is a sequence from the posterior simulator using dataY1:t .

16 / 33

Results

Data

For the evaluation of forecasts, we use the real-time data setconstructed by Del Negro and Schorfheide (2012).

I Forecast horizons and data vintages are aligned with BlueChip survey publication dates.

I Output growth, Inflation, Federal Funds rate.

I Generate forecasts four times a year (January, April, July, andOctober).

To evaluate forecasts we recursively estimate DSGE models overthe 78 vintages starting from January 1992 to April 2011.

I All estimation samples start in 1964.

I Compute forecast errors based on actuals that are obtainedfrom the most recent vintage.

18 / 33

Data

Data, 1964Q2-2011Q1

19 / 33

Extracted Volatility

Stochastic Volatility, 1964Q2-2010Q3

Standard deviations of the structural shocks (Posterior mean):

I Dotted : Linear DSGE with constant volatility

I Solid : Linear DSGE with SV

20 / 33

Point Forecasts Evaluation: RMSEs

Linear vs Linear+SV, 1991Q4-2011Q1

I SV improves point forecasts for output growth.

I Difference in RMSEs for output growth is significant (Dieboldand Mariano, 1995)

21 / 33

Point Forecasts Evaluation: RMSEs

Quadratic vs Quadratic+SV, 1991Q4-2011Q1

I SV improves point forecasts for output growth for Quadraticmodel as well.

22 / 33

Point Forecasts Evaluation: RMSEs

Linear+SV vs Quadratic+SV, 1991Q4-2011Q1

I Quadratic term does not improve point forecasts.

I Similar results as in Pichler (2008) but with SVs.

23 / 33

Interval Forecasts: Coverage Rate

Coverage Rate of 70% Interval Forecasts, 1991Q4-2011Q1

If the interval forecast has a correct coverage rate, then it should be around 0.7line (dotted black line).

I Interval forecasts from the models with SV are closer to 0.7 line.

I Quadratic+SV has better coverage rates for inflation rate h ≥ 3.

24 / 33

Interval Forecasts

Output Growth Interval Forecasts (h = 1), 1991Q4-2011Q1

I In general, interval forecasts are shorter for the model with SV.

I pre-Great moderation sample effect.

I Rolling window estimation (with 80Q) helps but notmuch.

25 / 33

Interval Forecasts

Output Growth Interval Forecasts (h = 1), 1991Q4-2011Q1

I In general, interval forecasts are shorter for the model with SV.

I pre-Great moderation sample effect.

I Rolling window estimation (with 80Q) helps but notmuch.

26 / 33

Density Forecasts Evaluation: PITs, (h = 1)

PITs, 1-Step-Ahead Prediction, 1991Q4-2011Q1

Linear DSGE Model

Linear+SV

PITs are grouped into five equally sized bids. Under a uniform distribution,

each bin should contain 20% of the PITs, indicated by the solid horizontal lines

in the figure.

27 / 33

Density Forecasts Evaluation: PITs (h = 1)

PITs, 1-Step-Ahead Prediction, 1991Q4-2011Q1

Quadratic DSGE Model

Quadratic + SV

PITs are grouped into five equally sized bids. Under a uniform distribution,

each bin should contain 20% of the PITs, indicated by the solid horizontal lines

in the figure.

28 / 33

Density Forecasts Evaluation: PITs (h = 4)

PITs, 4-Step-Ahead Prediction, 1991Q4-2011Q1

Linear DSGE Model

Linear+SV

PITs are grouped into five equally sized bids. Under a uniform distribution,

each bin should contain 20% of the PITs, indicated by the solid horizontal lines

in the figure.

29 / 33

Density Forecasts Evaluation: PITs (h = 4)

PITs, 4-Step-Ahead Prediction, 1991Q4-2011Q1

Quadratic DSGE Model

Quadratic + SV

PITs are grouped into five equally sized bids. Under a uniform distribution,

each bin should contain 20% of the PITs, indicated by the solid horizontal lines

in the figure.

30 / 33

Density Forecasts Evaluation: Normalized Error

wi ,t = Φ−1(zi ,t)

LR Tests of normalized errors of 1-step ahead real-time forecasts

Std. Dev. (= 1) Mean (= 0) AR(1) coef. (= 0) LR test

(a) Output Growth

Linear 0.543 (0.000) 0.195 (0.005) 0.357 (0.021) 53.049 (0.000)Quadratic 0.547 (0.000) 0.206 (0.003) 0.364 (0.018) 52.984 (0.000)Linear+SV 0.827 (0.349) 0.093 (0.493) 0.254 (0.072) 10.610 (0.014)Quadratic+SV 0.845 (0.422) 0.146 (0.282) 0.315 (0.022) 13.547 (0.004)

(b) Inflation RateLinear 0.762 (0.316) 0.058 (0.597) -0.125 (0.474) 10.595 (0.014)Quadratic 0.738 (0.045) 0.075 (0.598) 0.119 (0.273) 12.694 (0.005)Linear+SV 0.868 (0.461) 0.130 (0.368) -0.000 (0.998) 3.915 (0.271)Quadratic+SV 0.890 (0.592) 0.050 (0.777) 0.090 (0.516) 2.569 (0.463)

(c) Fed Funds RateLinear 0.578 (0.000) -0.070 (0.658) 0.663 (0.000) 77.837 (0.000)Quadratic 0.615 (0.000) -0.067 (0.705) 0.672 (0.000) 73.064 (0.000)Linear+SV 0.866 (0.492) -0.119 (0.593) 0.744 (0.000) 66.514 (0.000)Quadratic+SV 0.859 (0.483) -0.073 (0.743) 0.712 (0.000) 58.611 (0.000)

31 / 33

Log Predictive Density

Log Predictive Score, 1991Q4-2011Q1

h = 1Q h = 2Q h = 4Q

Linear -3.99 -4.20 -4.91Quadratic -3.97 -4.49 -5.05Linear+SV -3.82 -4.66 -5.70Quadratic+SV -3.84 -4.63 -5.52

I Linear+SV performs the best for the 1-step-ahead prediction.

I Linear without SV performs better for h ≥ 2.

32 / 33

Conclusion

Summary

I Modelling the time variation improves the density forecastsespecially in the short-run.

I The second order perturbation method does not improveforecasts’ quality in general.

Future worksI More features

I Time-varying inflation target.I SV process: ARMA(p,q) as opposed to AR(1).

I Larger model

I Particle filter

33 / 33