![Page 1: Stat 565stat565.cwick.co.nz/lectures/07-estimation.pdf · Stat 565 Charlotte Wickham stat565.cwick.co.nz The PACF & Estimation for ARMA Jan 26 2016](https://reader034.vdocuments.net/reader034/viewer/2022051601/5ac44c4c7f8b9aa0518d568c/html5/thumbnails/1.jpg)
Stat 565
Charlotte Wickham stat565.cwick.co.nz
The PACF & Estimation for ARMA Jan 26 2016
![Page 2: Stat 565stat565.cwick.co.nz/lectures/07-estimation.pdf · Stat 565 Charlotte Wickham stat565.cwick.co.nz The PACF & Estimation for ARMA Jan 26 2016](https://reader034.vdocuments.net/reader034/viewer/2022051601/5ac44c4c7f8b9aa0518d568c/html5/thumbnails/2.jpg)
Which basic models might these simulated data come
from?AR(1), MA(1), or white noise
1
2 3
4 5
![Page 3: Stat 565stat565.cwick.co.nz/lectures/07-estimation.pdf · Stat 565 Charlotte Wickham stat565.cwick.co.nz The PACF & Estimation for ARMA Jan 26 2016](https://reader034.vdocuments.net/reader034/viewer/2022051601/5ac44c4c7f8b9aa0518d568c/html5/thumbnails/3.jpg)
Which is which?
ACFOne is AR(1), α1 = 0.6
The other is ARMA(1, 1), β1 = 0.5, α1 = 0.5
![Page 4: Stat 565stat565.cwick.co.nz/lectures/07-estimation.pdf · Stat 565 Charlotte Wickham stat565.cwick.co.nz The PACF & Estimation for ARMA Jan 26 2016](https://reader034.vdocuments.net/reader034/viewer/2022051601/5ac44c4c7f8b9aa0518d568c/html5/thumbnails/4.jpg)
Partial autocorrelation function
Basic idea: what is the correlation between xt and xt+h, after taking into account xt+1, xt+2, ..., xt+h-1?
Technically: Regress xt on xt+1, xt+2, ..., xt+h-1 to find the fitted value x̂t. Regress xt+h on xt+1, xt+2, ..., xt+h-1 to find the fitted value x̂t+h. Find cor(xt - x̂t, xt+h - x̂t+h), call this PACF(h) = ɸhh
Section 3.4 S&S for more detail
![Page 5: Stat 565stat565.cwick.co.nz/lectures/07-estimation.pdf · Stat 565 Charlotte Wickham stat565.cwick.co.nz The PACF & Estimation for ARMA Jan 26 2016](https://reader034.vdocuments.net/reader034/viewer/2022051601/5ac44c4c7f8b9aa0518d568c/html5/thumbnails/5.jpg)
MA(q)
MA(1): xt = wt +1/2 wt-1 MA(2): xt = wt + 1/6 wt-1 + 1/2 wt-2
MA(5): xt = wt - 1/2 wt-1 - 1/2 wt-2 +
1/4 wt-3 + 1/4 wt-4 +1/4 wt-5
MA(10), θj=1/2 j = 1,...,10
PACF
![Page 6: Stat 565stat565.cwick.co.nz/lectures/07-estimation.pdf · Stat 565 Charlotte Wickham stat565.cwick.co.nz The PACF & Estimation for ARMA Jan 26 2016](https://reader034.vdocuments.net/reader034/viewer/2022051601/5ac44c4c7f8b9aa0518d568c/html5/thumbnails/6.jpg)
AR(p)
AR(1): xt = wt +1/2 xt-1 AR(2): xt = wt + 1/6 xt-1 + 1/2 xt-2
AR(5): xt = wt - 1/2 xt-1 - 1/2 xt-2 +
1/4 xt-3 + 1/4 xt-4 +1/4 xt-5
AR(8), ɸj=1/9 j = 1,...,8
PACF
1
![Page 7: Stat 565stat565.cwick.co.nz/lectures/07-estimation.pdf · Stat 565 Charlotte Wickham stat565.cwick.co.nz The PACF & Estimation for ARMA Jan 26 2016](https://reader034.vdocuments.net/reader034/viewer/2022051601/5ac44c4c7f8b9aa0518d568c/html5/thumbnails/7.jpg)
ARMA(p, q)
ARMA(1, 1): xt = 1/2 wt-1 + wt +1/2 xt-1 ARMA(2, 1): xt = 1/2 wt-1 + wt +
1/6 xt-1 + 1/2 xt-2
ARMA(2,2): xt = 1/4wt-2 -1/2 wt-1 + wt -
1/2 xt-1 + 1/4 xt-2
ARMA(2,2): xt = -1/4wt-2 +1/2 wt-1 + wt +
1/9 xt-1 + 1/9 xt-2
PACF
![Page 8: Stat 565stat565.cwick.co.nz/lectures/07-estimation.pdf · Stat 565 Charlotte Wickham stat565.cwick.co.nz The PACF & Estimation for ARMA Jan 26 2016](https://reader034.vdocuments.net/reader034/viewer/2022051601/5ac44c4c7f8b9aa0518d568c/html5/thumbnails/8.jpg)
MA(q) AR(p) ARMA(p, q)
ACFzero
lags > q tails off tails off
PACF tails offzero lags >
p tails off
![Page 9: Stat 565stat565.cwick.co.nz/lectures/07-estimation.pdf · Stat 565 Charlotte Wickham stat565.cwick.co.nz The PACF & Estimation for ARMA Jan 26 2016](https://reader034.vdocuments.net/reader034/viewer/2022051601/5ac44c4c7f8b9aa0518d568c/html5/thumbnails/9.jpg)
Which is which?
One is AR(1), α1 = 0.6The other is ARMA(1, 1), β1 = 0.5, α1 = 0.5
ACF
PACF
![Page 10: Stat 565stat565.cwick.co.nz/lectures/07-estimation.pdf · Stat 565 Charlotte Wickham stat565.cwick.co.nz The PACF & Estimation for ARMA Jan 26 2016](https://reader034.vdocuments.net/reader034/viewer/2022051601/5ac44c4c7f8b9aa0518d568c/html5/thumbnails/10.jpg)
Corvallis temperature
ACF of residuals
![Page 11: Stat 565stat565.cwick.co.nz/lectures/07-estimation.pdf · Stat 565 Charlotte Wickham stat565.cwick.co.nz The PACF & Estimation for ARMA Jan 26 2016](https://reader034.vdocuments.net/reader034/viewer/2022051601/5ac44c4c7f8b9aa0518d568c/html5/thumbnails/11.jpg)
Corvallis temperature
PACF of residuals
AR(1) looks like a good model
How do we fit it? I.e. how do we estimate α1?
pacf(corv$residual, na.action = na.pass)
![Page 12: Stat 565stat565.cwick.co.nz/lectures/07-estimation.pdf · Stat 565 Charlotte Wickham stat565.cwick.co.nz The PACF & Estimation for ARMA Jan 26 2016](https://reader034.vdocuments.net/reader034/viewer/2022051601/5ac44c4c7f8b9aa0518d568c/html5/thumbnails/12.jpg)
ARMA(p,q)
Assume we know the order of our process, i.e. we know p and q.
How do we estimate the βs, αs, and σ2?
![Page 13: Stat 565stat565.cwick.co.nz/lectures/07-estimation.pdf · Stat 565 Charlotte Wickham stat565.cwick.co.nz The PACF & Estimation for ARMA Jan 26 2016](https://reader034.vdocuments.net/reader034/viewer/2022051601/5ac44c4c7f8b9aa0518d568c/html5/thumbnails/13.jpg)
Your turn
Name three common approaches to finding an estimate:Hints:M of ML S M L
The default in R Has nice properties
![Page 14: Stat 565stat565.cwick.co.nz/lectures/07-estimation.pdf · Stat 565 Charlotte Wickham stat565.cwick.co.nz The PACF & Estimation for ARMA Jan 26 2016](https://reader034.vdocuments.net/reader034/viewer/2022051601/5ac44c4c7f8b9aa0518d568c/html5/thumbnails/14.jpg)
We have the theoretical ACF in terms of α, β and σ. Set the theoretical ACF equal to our sample ACF and solve for the parameters.
M_____ __ M________
So, we don't get too confused, let ρ(h) denote the theoretical ACF, and r(h) the sample ACF.
![Page 15: Stat 565stat565.cwick.co.nz/lectures/07-estimation.pdf · Stat 565 Charlotte Wickham stat565.cwick.co.nz The PACF & Estimation for ARMA Jan 26 2016](https://reader034.vdocuments.net/reader034/viewer/2022051601/5ac44c4c7f8b9aa0518d568c/html5/thumbnails/15.jpg)
Corvallis temperature
Assume the residuals can be modelled by an AR(1). ρ(1) = ɸ r(1) = = 0.6607
ɸ = α, notation slip
![Page 16: Stat 565stat565.cwick.co.nz/lectures/07-estimation.pdf · Stat 565 Charlotte Wickham stat565.cwick.co.nz The PACF & Estimation for ARMA Jan 26 2016](https://reader034.vdocuments.net/reader034/viewer/2022051601/5ac44c4c7f8b9aa0518d568c/html5/thumbnails/16.jpg)
AR(p)
For AR(p) processes we write down a recursion,
The Yule-Walker equations
![Page 17: Stat 565stat565.cwick.co.nz/lectures/07-estimation.pdf · Stat 565 Charlotte Wickham stat565.cwick.co.nz The PACF & Estimation for ARMA Jan 26 2016](https://reader034.vdocuments.net/reader034/viewer/2022051601/5ac44c4c7f8b9aa0518d568c/html5/thumbnails/17.jpg)
Derive Yule-Walker eqns
![Page 18: Stat 565stat565.cwick.co.nz/lectures/07-estimation.pdf · Stat 565 Charlotte Wickham stat565.cwick.co.nz The PACF & Estimation for ARMA Jan 26 2016](https://reader034.vdocuments.net/reader034/viewer/2022051601/5ac44c4c7f8b9aa0518d568c/html5/thumbnails/18.jpg)
Yule-Walker Estimates
A set of p equations in p unknowns, solve for to .
![Page 19: Stat 565stat565.cwick.co.nz/lectures/07-estimation.pdf · Stat 565 Charlotte Wickham stat565.cwick.co.nz The PACF & Estimation for ARMA Jan 26 2016](https://reader034.vdocuments.net/reader034/viewer/2022051601/5ac44c4c7f8b9aa0518d568c/html5/thumbnails/19.jpg)
MA(q) and ARMA(p,q)
The method of moments approach gets complicated. End up with non-linear equations to be solved numerically.
The method of moments estimators have bad properties for MA and ARMA processes anyway, so we'll leave it here.
![Page 20: Stat 565stat565.cwick.co.nz/lectures/07-estimation.pdf · Stat 565 Charlotte Wickham stat565.cwick.co.nz The PACF & Estimation for ARMA Jan 26 2016](https://reader034.vdocuments.net/reader034/viewer/2022051601/5ac44c4c7f8b9aa0518d568c/html5/thumbnails/20.jpg)
Your turnRemember linear regression? yi = β0 + β1xi + εiWe can find estimates for β0 ,β1 by minimizing
What goes in the .... ?
nX
i=1
⇣yi � �̂0 � �̂1xi
⌘2
![Page 21: Stat 565stat565.cwick.co.nz/lectures/07-estimation.pdf · Stat 565 Charlotte Wickham stat565.cwick.co.nz The PACF & Estimation for ARMA Jan 26 2016](https://reader034.vdocuments.net/reader034/viewer/2022051601/5ac44c4c7f8b9aa0518d568c/html5/thumbnails/21.jpg)
L______ s_______
Consider the AR(1) process xt = α1xt-1 + wt Define the residual, We could consider finding the that minimises the sum of squared residuals,
since we don't see x0
called conditional least squares
et = xt � ↵̂1xt�1
↵̂1
nX
t=2
(xt � ↵̂1xt�1)2 =
nX
t=2
e
2t
![Page 22: Stat 565stat565.cwick.co.nz/lectures/07-estimation.pdf · Stat 565 Charlotte Wickham stat565.cwick.co.nz The PACF & Estimation for ARMA Jan 26 2016](https://reader034.vdocuments.net/reader034/viewer/2022051601/5ac44c4c7f8b9aa0518d568c/html5/thumbnails/22.jpg)
L_____ s_____ for MA
In general we can always define these residuals, but for MA and ARMA processes they are recursive. For example, MA(1)
assume e0 = 0e1 = x1 + �1e0
e2 = x2 + �1e1
...
en = xn + �1en�1
![Page 23: Stat 565stat565.cwick.co.nz/lectures/07-estimation.pdf · Stat 565 Charlotte Wickham stat565.cwick.co.nz The PACF & Estimation for ARMA Jan 26 2016](https://reader034.vdocuments.net/reader034/viewer/2022051601/5ac44c4c7f8b9aa0518d568c/html5/thumbnails/23.jpg)
L_____ s______ in ARMA
For a general ARMA(p,q):
And we have to set ep = ... = ep+1-q = 0, and sum starting at t = p, to avoid the xt we haven't observed
The minimization is done numerically
et = xt � ↵1xt�1 � . . .� ↵pxt�p
+ �1et�1 + . . .+ �qet�q
![Page 24: Stat 565stat565.cwick.co.nz/lectures/07-estimation.pdf · Stat 565 Charlotte Wickham stat565.cwick.co.nz The PACF & Estimation for ARMA Jan 26 2016](https://reader034.vdocuments.net/reader034/viewer/2022051601/5ac44c4c7f8b9aa0518d568c/html5/thumbnails/24.jpg)
M________ L_________Assume a distribution for the white noise (usually Gaussian), then write the joint density function of our data as a function of the parameters, the likelihood, L(β, θ, σ2) = f(x1, x2, ... xn; β, θ, σ2) Find the parameters that maximise the likelihood.
For the non-statisticans: The joint density, f, tells us the probability of our data given certain parameter values. The likelihood, L, tells us how likely certain parameters are given our data (f and L are the same function, we just switch what we consider to be the variable). We estimate the parameters by choosing the most likely parameters given the data we saw.
![Page 25: Stat 565stat565.cwick.co.nz/lectures/07-estimation.pdf · Stat 565 Charlotte Wickham stat565.cwick.co.nz The PACF & Estimation for ARMA Jan 26 2016](https://reader034.vdocuments.net/reader034/viewer/2022051601/5ac44c4c7f8b9aa0518d568c/html5/thumbnails/25.jpg)
M________ L_________
Assuming our white noise is Gaussian then the ARMA(p, q), xt, process is also Gaussian and the likelihood is
where
It's complicated, but there are general algorithms for maximizing it.
![Page 26: Stat 565stat565.cwick.co.nz/lectures/07-estimation.pdf · Stat 565 Charlotte Wickham stat565.cwick.co.nz The PACF & Estimation for ARMA Jan 26 2016](https://reader034.vdocuments.net/reader034/viewer/2022051601/5ac44c4c7f8b9aa0518d568c/html5/thumbnails/26.jpg)
Maximum Likelihood
The way the function arima in R does it by default. Nice asymptotic properties, deals with missing data easily. Always lower variance than method of moments.
![Page 27: Stat 565stat565.cwick.co.nz/lectures/07-estimation.pdf · Stat 565 Charlotte Wickham stat565.cwick.co.nz The PACF & Estimation for ARMA Jan 26 2016](https://reader034.vdocuments.net/reader034/viewer/2022051601/5ac44c4c7f8b9aa0518d568c/html5/thumbnails/27.jpg)
Fitting in R
arima(ts, order = c(p, d, q))
ignore for now
![Page 28: Stat 565stat565.cwick.co.nz/lectures/07-estimation.pdf · Stat 565 Charlotte Wickham stat565.cwick.co.nz The PACF & Estimation for ARMA Jan 26 2016](https://reader034.vdocuments.net/reader034/viewer/2022051601/5ac44c4c7f8b9aa0518d568c/html5/thumbnails/28.jpg)
Thursday
I’m out of town. Chris will lead lecture. Bring laptops!