cy3a2 system identification1 maximum likelihood estimation: maximum likelihood is an ancient concept...
TRANSCRIPT
CY3A2 System identification 1
Maximum Likelihood Estimation:
Maximum Likelihood is an ancient concept in estimation theory.
Suppose that e is a discrete random process, and we know its probability density function as a functional of a parameter θ, such that we know P(e; θ).
Now we have n data samples, given just as before ( y, u ), how do we estimate θ ?
The idea of Maximum Likelihood Estimation is to maximize a Likelihood function which is often defined as the joint probability of ei.
CY3A2 System identification 2
Suppose ei is uncorrelated, the Likelihood function L can be written as (the joint probability of ei)
(1)
n
iin,ep;e,...e,eL
111
This means that the Likelihood function is the product of data each sample’s pdf.
Consider using log Likelihood function Log L.
Log function is a monotonous function. This means when
L is maximum, so is Log L.
CY3A2 System identification 3
(1) loglog
n
iin,ep;e,...e,eL
111
Instead of looking for , that maximizes L,We now look for , that maximizes log L, the result will be the same, but computation is simpler!
(2)
log011
ˆ
n;e,...e,eL
CY3A2 System identification 4
If is Gaussian with zero mean, and variance i
e 2
),(N~ei
20
iii
yeAlso consider the link between and data observations is
ie
2
2
2
2
2
2
22
1
22
1
Tii
ii
y
e,ep
exp
exp
CY3A2 System identification 5
exp
loglog
c
logn
ylog
,ep;e,...e,eL
T
T
n
i
Tii
n
iin
2
22
12
2
2
111
2
222
22
1
yy
yy
CY3A2 System identification 6
(2) log
0
ˆ
L
By setting
We get 0
yyT
yTTˆ 1
Which is simply equivalent to LS estimate.
A common fact: Under Gaussian assumption, the Least Squares estimates is equivalent to Maximum Likelihood estimate.
CY3A2 System identification 7
Modelling Nonlinear AutoRegressive (NAR) Model by Radial Basis Function (RBF) neural networks
inyiiiie)y,...y,y(fy
21
ik
kki
ey
T
nyiii
ki
k
y,y
c
1
2
2
x
xexp
e.g Gaussian Radial basis function:
CY3A2 System identification 8
Radial Basis Function Neural Networks
CY3A2 System identification 9
Least squares (LS) can be readily used to identify RBF networks.
1. Some method to determine the centres (k-means clustering, or random selection from the data set), and given width σ.2. You know how to estimate θ.
yTTˆ 1
2
2
ki
k
cxexp
is filled by