point, interval, and density forecast evaluation of linear...
TRANSCRIPT
Point, Interval, and Density Forecast Evaluationof Linear versus Nonlinear DSGE Models
Francis X. Diebold Frank Schorfheide Minchul Shin
University of Pennsylvania
May 4, 2014
1 / 33
Motivation
The use of DSGE models as a forecasting tool in practice calls forevaluation of its performance.
Summary of DSGE forecasting evaluation literature:
I Point forecasts are as good as other statistical models such asVAR.
I Density forecasts are too wide.
Typically, DSGE models in this literature are
I Solved with linearized method.
I Constant volatility.
Forecasting with a nonlinear DSGE model:
I Pichler (2008) considers the second-order perturbationmethod.
2 / 33
Nonlinearity and Forecasting
Not much work has been done for nonlinear DSGE forecasting.
Two types of nonlinearitiesI Time-varying volatility vs. Constant volatility
I There are many papers on DSGE models with time-varyingvolatility but there is no paper that systematically evaluatesforecasts.
I Most papers report that density forecasts generated from theDSGE models with constant volatility is too wide.
I Nonlinear approximation vs. Linear approximationI Pichler (2008)’s point forecast evaluation.I Interval and density forecasts evaluation.
3 / 33
Summary
MethodI Small-scale DSGE model.
I Linear DSGE model with constant volatility.I Linear DSGE model with time-varying volatility.I Quadratic DSGE model (based on the second-order perturbation
method).I Quadratic DSGE model with time-varying volatility.
I Bayesian estimation and forecasting. (US data, 1964-2011)
Finding
I Modelling time-varying volatility improves accuracy offorecasts.
I Second-order perturbation method does not improve.
4 / 33
Model, Estimation, and Forecasting
Small-Scale DSGE Model
Small-scale model used in Herbst and Schorfheide (2012)
I Euler equation, NK Phillips curve, Monetary policy rule
I 3 exogenous shocks: technology, government spending,monetary policy (zt , gt ,mpt)
Measurement equations:(YGRt
INFt
FFRt
)= D(θ) + Z (θ) st
Transition equations:
st = Φ(st−1, εt ; θ)
where
st = [yt , yt−1, ct , πt ,Rt ,mpt , zt , gt ]′
εt : Innovations
θ : DSGE parameters
6 / 33
Linear DSGE Model
Linear approximation methods lead to a linear Gaussian state spacerepresentation with the following transition equation
st = H(θ)st−1 + R(θ)εt , εt ∼ iidN (0,Q(θ)).
I Coefficient matrices (H(θ),R(θ),Q(θ)) are the nonlinearfunction of θ.
I We obtain posterior draws based on the Random WalkMetropolis (RWM) algorithm with the Kalman filter.
7 / 33
Linear+Stochastic Volatility
Following Justiniano and Primiceri (2008),
st = H(θ)st−1 + R(θ)εt , εt ∼ N (0,Qt(θ))
where
diag(Qt(θ)) = [e2hmp,t , e2hz,t , e2hg,t ]′
hi ,t = ρσihi ,t−1 + νi ,t , νi ,t ∼ iidN (0, s2i ),
for i = mp, g , z .
I The system is in a linear Gaussian state-space formconditional on Qt(θ).
I We use the Metropolis-within-Gibbs algorithm developed byKim, Shephard, and Chib (1998) to generate draws from theposterior distribution.
8 / 33
Quadratic DSGE Model
The equilibrium law of motion based on the second-orderperturbation method has the following form:
s1,t = G0(θ) + G1(θ)s2,t + G2(θ)(s2,t ⊗ s2,t)
s2,t = H0(θ) + H1(θ)s2,t−1 + H2(θ)(s2,t−1 ⊗ s2,t−1) + R(θ)εt
where εt ∼ iidN (0,Q(θ)), st = [s1,t , s2,t ]′ and ⊗ is a Kronecker
product.
I We have an additional quadratic term.
I We need a nonlinear filter to get posterior draws.
I Particle filter + RWM (exact).I Second-order Extended Kalman Filter + RWM (approximate).
I Current version of the paper utilizes the second-order extended Kalmanfilter and RWM.
I We also consider Quadratic DSGE model with stochastic volatility.
9 / 33
Predictive Distribution
We generate draws from the posterior predictive distribution basedon the following decomposition,
p(YT+1:T+H |Y1:T ) =
∫p(YT+1:T+H |θ,Y1:T )p(θ|Y1:T )dθ
I p(θ|Y1:T ) : Posterior sampler.
I p(YT+1:T+H |θ,Y1:T ) : Given θ, simulate the model economyforward.
Draws {Y (j)T+1:T+H}
nsimj=1 can be turned into point and interval
forecasts by the Monte Carlo average,
E[yT+h|T
]=
∫yT+h
yT+h p(yT+h|Y1:T )dyT+h ≈1
nsim
nsim∑j=1
y(j)T+h.
10 / 33
Evaluation Methods
Point Forecast
Point forecasts from two different models (Inflation rate)
We compare RMSEs for variable i = {YGR, INFL, FFR},
RMSE(i |h) =
√√√√ 1
P − h
R+P−h∑t=R
(yi,t+h − yi,t+h|t)2
where R is the index denotes the starting point of the forecasts evaluationsample and P the number of forecasting origins.
12 / 33
Interval Forecast
Interval forecasts from two different models (Inflation rate)
I We compute α% interval forecasts for a particular element yi,T+h of yT+h
by numerically searching for the shortest connected interval that containsa α% of the draws {y (j)
i,T+h}nsimj=1 (the highest-density set).
I If the interval forecast is well calibrated, actual variables are expected tobe inside of α% interval forecasts at the same frequency.
13 / 33
Interval Forecast
Interval forecasts from two different models (Inflation rate)
Christoffersen (1998)’s LR tests of the correct coverage (α):
I Define the sequence of hit indicators of a 1-step-ahead forecast interval,
Iαt = 1{realized yt falls inside the interval}
I TestIαt ∼ iidBernoulli(α).
14 / 33
Density Forecast
Density forecasts from two different models (Inflation rate)
The probability integral transform (PIT) of yi,T+h based on time T predictivedistribution is defined as the cumulative density of the random variable,
zi,h,T =
∫ yi,T+h
−∞p(yi,T+h|Y1:T )dyi,T+h.
I If the predictive distribution is well-calibrated, zi,h,T should follow theuniform distribution.
I For h = 1, zi,h,T ’s follow independent uniformly distribution.
I Diebold, Gunther, and Tay (1998).
15 / 33
Density Forecast Evaluation: Predictive Likelihood
The one-step-ahead predictive likelihood,
PL(t) = p(yt+1|Y1:t)
I Height of the predictive density at the realized value yt+1.
I Log predictive score:∑R+P−1
t=R logPL(t).
We approximate
PL(t) ≈ 1
M
M∑m=1
p(yt+1|Y1:t , θ
(m))
where {θ}(m) is a sequence from the posterior simulator using dataY1:t .
16 / 33
Results
Data
For the evaluation of forecasts, we use the real-time data setconstructed by Del Negro and Schorfheide (2012).
I Forecast horizons and data vintages are aligned with BlueChip survey publication dates.
I Output growth, Inflation, Federal Funds rate.
I Generate forecasts four times a year (January, April, July, andOctober).
To evaluate forecasts we recursively estimate DSGE models overthe 78 vintages starting from January 1992 to April 2011.
I All estimation samples start in 1964.
I Compute forecast errors based on actuals that are obtainedfrom the most recent vintage.
18 / 33
Data
Data, 1964Q2-2011Q1
19 / 33
Extracted Volatility
Stochastic Volatility, 1964Q2-2010Q3
Standard deviations of the structural shocks (Posterior mean):
I Dotted : Linear DSGE with constant volatility
I Solid : Linear DSGE with SV
20 / 33
Point Forecasts Evaluation: RMSEs
Linear vs Linear+SV, 1991Q4-2011Q1
I SV improves point forecasts for output growth.
I Difference in RMSEs for output growth is significant (Dieboldand Mariano, 1995)
21 / 33
Point Forecasts Evaluation: RMSEs
Quadratic vs Quadratic+SV, 1991Q4-2011Q1
I SV improves point forecasts for output growth for Quadraticmodel as well.
22 / 33
Point Forecasts Evaluation: RMSEs
Linear+SV vs Quadratic+SV, 1991Q4-2011Q1
I Quadratic term does not improve point forecasts.
I Similar results as in Pichler (2008) but with SVs.
23 / 33
Interval Forecasts: Coverage Rate
Coverage Rate of 70% Interval Forecasts, 1991Q4-2011Q1
If the interval forecast has a correct coverage rate, then it should be around 0.7line (dotted black line).
I Interval forecasts from the models with SV are closer to 0.7 line.
I Quadratic+SV has better coverage rates for inflation rate h ≥ 3.
24 / 33
Interval Forecasts
Output Growth Interval Forecasts (h = 1), 1991Q4-2011Q1
I In general, interval forecasts are shorter for the model with SV.
I pre-Great moderation sample effect.
I Rolling window estimation (with 80Q) helps but notmuch.
25 / 33
Interval Forecasts
Output Growth Interval Forecasts (h = 1), 1991Q4-2011Q1
I In general, interval forecasts are shorter for the model with SV.
I pre-Great moderation sample effect.
I Rolling window estimation (with 80Q) helps but notmuch.
26 / 33
Density Forecasts Evaluation: PITs, (h = 1)
PITs, 1-Step-Ahead Prediction, 1991Q4-2011Q1
Linear DSGE Model
Linear+SV
PITs are grouped into five equally sized bids. Under a uniform distribution,
each bin should contain 20% of the PITs, indicated by the solid horizontal lines
in the figure.
27 / 33
Density Forecasts Evaluation: PITs (h = 1)
PITs, 1-Step-Ahead Prediction, 1991Q4-2011Q1
Quadratic DSGE Model
Quadratic + SV
PITs are grouped into five equally sized bids. Under a uniform distribution,
each bin should contain 20% of the PITs, indicated by the solid horizontal lines
in the figure.
28 / 33
Density Forecasts Evaluation: PITs (h = 4)
PITs, 4-Step-Ahead Prediction, 1991Q4-2011Q1
Linear DSGE Model
Linear+SV
PITs are grouped into five equally sized bids. Under a uniform distribution,
each bin should contain 20% of the PITs, indicated by the solid horizontal lines
in the figure.
29 / 33
Density Forecasts Evaluation: PITs (h = 4)
PITs, 4-Step-Ahead Prediction, 1991Q4-2011Q1
Quadratic DSGE Model
Quadratic + SV
PITs are grouped into five equally sized bids. Under a uniform distribution,
each bin should contain 20% of the PITs, indicated by the solid horizontal lines
in the figure.
30 / 33
Density Forecasts Evaluation: Normalized Error
wi ,t = Φ−1(zi ,t)
LR Tests of normalized errors of 1-step ahead real-time forecasts
Std. Dev. (= 1) Mean (= 0) AR(1) coef. (= 0) LR test
(a) Output Growth
Linear 0.543 (0.000) 0.195 (0.005) 0.357 (0.021) 53.049 (0.000)Quadratic 0.547 (0.000) 0.206 (0.003) 0.364 (0.018) 52.984 (0.000)Linear+SV 0.827 (0.349) 0.093 (0.493) 0.254 (0.072) 10.610 (0.014)Quadratic+SV 0.845 (0.422) 0.146 (0.282) 0.315 (0.022) 13.547 (0.004)
(b) Inflation RateLinear 0.762 (0.316) 0.058 (0.597) -0.125 (0.474) 10.595 (0.014)Quadratic 0.738 (0.045) 0.075 (0.598) 0.119 (0.273) 12.694 (0.005)Linear+SV 0.868 (0.461) 0.130 (0.368) -0.000 (0.998) 3.915 (0.271)Quadratic+SV 0.890 (0.592) 0.050 (0.777) 0.090 (0.516) 2.569 (0.463)
(c) Fed Funds RateLinear 0.578 (0.000) -0.070 (0.658) 0.663 (0.000) 77.837 (0.000)Quadratic 0.615 (0.000) -0.067 (0.705) 0.672 (0.000) 73.064 (0.000)Linear+SV 0.866 (0.492) -0.119 (0.593) 0.744 (0.000) 66.514 (0.000)Quadratic+SV 0.859 (0.483) -0.073 (0.743) 0.712 (0.000) 58.611 (0.000)
31 / 33
Log Predictive Density
Log Predictive Score, 1991Q4-2011Q1
h = 1Q h = 2Q h = 4Q
Linear -3.99 -4.20 -4.91Quadratic -3.97 -4.49 -5.05Linear+SV -3.82 -4.66 -5.70Quadratic+SV -3.84 -4.63 -5.52
I Linear+SV performs the best for the 1-step-ahead prediction.
I Linear without SV performs better for h ≥ 2.
32 / 33
Conclusion
Summary
I Modelling the time variation improves the density forecastsespecially in the short-run.
I The second order perturbation method does not improveforecasts’ quality in general.
Future worksI More features
I Time-varying inflation target.I SV process: ARMA(p,q) as opposed to AR(1).
I Larger model
I Particle filter
33 / 33