least square fit of a line/polynomial to input/output data · 2015. 7. 16. · least square t of a...
TRANSCRIPT
![Page 1: Least square fit of a line/polynomial to input/output data · 2015. 7. 16. · Least square t of a line/polynomial to input/output data Prashant Dave Chemical Engg., Indian Institute](https://reader035.vdocuments.net/reader035/viewer/2022071514/613585b80ad5d20676476e54/html5/thumbnails/1.jpg)
Least square fit of a line/polynomial toinput/output data
Prashant Dave
Chemical Engg.,Indian Institute of Technology Bombay
Jan, 2012
![Page 2: Least square fit of a line/polynomial to input/output data · 2015. 7. 16. · Least square t of a line/polynomial to input/output data Prashant Dave Chemical Engg., Indian Institute](https://reader035.vdocuments.net/reader035/viewer/2022071514/613585b80ad5d20676476e54/html5/thumbnails/2.jpg)
Outline Scilab Least squares
Outline
1 Scilab
2 Least squares
Prashant Dave Least square fit
![Page 3: Least square fit of a line/polynomial to input/output data · 2015. 7. 16. · Least square t of a line/polynomial to input/output data Prashant Dave Chemical Engg., Indian Institute](https://reader035.vdocuments.net/reader035/viewer/2022071514/613585b80ad5d20676476e54/html5/thumbnails/3.jpg)
Outline Scilab Least squares
Today’s focus
Scilab is free.
Matrix/loops syntax is same as for Matlab.
Scilab provides all basic and many advanced tools.
Today: best fit: line and polynomial : reglin command
Prashant Dave Least square fit
![Page 4: Least square fit of a line/polynomial to input/output data · 2015. 7. 16. · Least square t of a line/polynomial to input/output data Prashant Dave Chemical Engg., Indian Institute](https://reader035.vdocuments.net/reader035/viewer/2022071514/613585b80ad5d20676476e54/html5/thumbnails/4.jpg)
Outline Scilab Least squares
Today’s focus
Scilab is free.
Matrix/loops syntax is same as for Matlab.
Scilab provides all basic and many advanced tools.
Today: best fit: line and polynomial : reglin command
Prashant Dave Least square fit
![Page 5: Least square fit of a line/polynomial to input/output data · 2015. 7. 16. · Least square t of a line/polynomial to input/output data Prashant Dave Chemical Engg., Indian Institute](https://reader035.vdocuments.net/reader035/viewer/2022071514/613585b80ad5d20676476e54/html5/thumbnails/5.jpg)
Outline Scilab Least squares
Today’s focus
Scilab is free.
Matrix/loops syntax is same as for Matlab.
Scilab provides all basic and many advanced tools.
Today: best fit: line and polynomial : reglin command
Prashant Dave Least square fit
![Page 6: Least square fit of a line/polynomial to input/output data · 2015. 7. 16. · Least square t of a line/polynomial to input/output data Prashant Dave Chemical Engg., Indian Institute](https://reader035.vdocuments.net/reader035/viewer/2022071514/613585b80ad5d20676476e54/html5/thumbnails/6.jpg)
Outline Scilab Least squares
Today’s focus
Scilab is free.
Matrix/loops syntax is same as for Matlab.
Scilab provides all basic and many advanced tools.
Today: best fit: line and polynomial : reglin command
Prashant Dave Least square fit
![Page 7: Least square fit of a line/polynomial to input/output data · 2015. 7. 16. · Least square t of a line/polynomial to input/output data Prashant Dave Chemical Engg., Indian Institute](https://reader035.vdocuments.net/reader035/viewer/2022071514/613585b80ad5d20676476e54/html5/thumbnails/7.jpg)
Outline Scilab Least squares
Linear fit
Given n samples of (x , y) pairs:xi and yi for i = 1, . . . , n, we expect following equation is satisfied
yi = a1xi + a0 for i = 1, . . . , n (1)
for some constants a1 and a0.
x: independent variable (exactly known),y: dependent variable (some error in measuring it)xi and yi fall on some line with slope a1 and ‘y-intercept’=a0.The ‘line fit’ problem:Find these constants a1 and a0.‘Best’ fit?
Prashant Dave Least square fit
![Page 8: Least square fit of a line/polynomial to input/output data · 2015. 7. 16. · Least square t of a line/polynomial to input/output data Prashant Dave Chemical Engg., Indian Institute](https://reader035.vdocuments.net/reader035/viewer/2022071514/613585b80ad5d20676476e54/html5/thumbnails/8.jpg)
Outline Scilab Least squares
Linear fit
Given n samples of (x , y) pairs:xi and yi for i = 1, . . . , n, we expect following equation is satisfied
yi = a1xi + a0 for i = 1, . . . , n (1)
for some constants a1 and a0.x: independent variable (exactly known),y: dependent variable (some error in measuring it)xi and yi fall on some line with slope a1 and ‘y-intercept’=a0.The ‘line fit’ problem:Find these constants a1 and a0.‘Best’ fit?
Prashant Dave Least square fit
![Page 9: Least square fit of a line/polynomial to input/output data · 2015. 7. 16. · Least square t of a line/polynomial to input/output data Prashant Dave Chemical Engg., Indian Institute](https://reader035.vdocuments.net/reader035/viewer/2022071514/613585b80ad5d20676476e54/html5/thumbnails/9.jpg)
Outline Scilab Least squares
Best fit
The true relationship is yi = a0a + a1axi , but due to noise (forexample in measurements), the available xi , yi pairs will not satisfythe equation exactly.
Least-square-fit problem:Given n samples of (xi , yi ) pairs,find constants a1 and a0 such that the ‘total square error’
n∑i=1
(yi − a1xi − a0)2 (2)
is least.
Prashant Dave Least square fit
![Page 10: Least square fit of a line/polynomial to input/output data · 2015. 7. 16. · Least square t of a line/polynomial to input/output data Prashant Dave Chemical Engg., Indian Institute](https://reader035.vdocuments.net/reader035/viewer/2022071514/613585b80ad5d20676476e54/html5/thumbnails/10.jpg)
Outline Scilab Least squares
Best fit
The true relationship is yi = a0a + a1axi , but due to noise (forexample in measurements), the available xi , yi pairs will not satisfythe equation exactly.Least-square-fit problem:Given n samples of (xi , yi ) pairs,
find constants a1 and a0 such that the ‘total square error’
n∑i=1
(yi − a1xi − a0)2 (2)
is least.
Prashant Dave Least square fit
![Page 11: Least square fit of a line/polynomial to input/output data · 2015. 7. 16. · Least square t of a line/polynomial to input/output data Prashant Dave Chemical Engg., Indian Institute](https://reader035.vdocuments.net/reader035/viewer/2022071514/613585b80ad5d20676476e54/html5/thumbnails/11.jpg)
Outline Scilab Least squares
Best fit
The true relationship is yi = a0a + a1axi , but due to noise (forexample in measurements), the available xi , yi pairs will not satisfythe equation exactly.Least-square-fit problem:Given n samples of (xi , yi ) pairs,find constants a1 and a0 such that the ‘total square error’
n∑i=1
(yi − a1xi − a0)2 (2)
is least.
Prashant Dave Least square fit
![Page 12: Least square fit of a line/polynomial to input/output data · 2015. 7. 16. · Least square t of a line/polynomial to input/output data Prashant Dave Chemical Engg., Indian Institute](https://reader035.vdocuments.net/reader035/viewer/2022071514/613585b80ad5d20676476e54/html5/thumbnails/12.jpg)
Outline Scilab Least squares
Scilab Tool: reglin
[a1,a0,sig]=reglin(x,y)
x: 1 × n vector (for n data points)
y: 1 × n vector (for n data points)
a1: slope, a0: intercept
sig: standard deviation of fit error: lower is “better”
Prashant Dave Least square fit
![Page 13: Least square fit of a line/polynomial to input/output data · 2015. 7. 16. · Least square t of a line/polynomial to input/output data Prashant Dave Chemical Engg., Indian Institute](https://reader035.vdocuments.net/reader035/viewer/2022071514/613585b80ad5d20676476e54/html5/thumbnails/13.jpg)
Outline Scilab Least squares
Straight line fit example
Generate data using known (actual) values of a0 and a1.Add noise to dependent variable.Using noisy data, estimate a0 and a1.
1 True data generation: y = 5 + 2x for x = 0 : 10.
2 Noise addition: y = y + e where e is normally distributednoise with mean 0 and standard deviation 2.
3 Least squares fit: [a1, a0, sig ] = reglin(x , y).
4 Plot: (xi,yi) pairs, true (noise free) line, fitted line
Prashant Dave Least square fit
![Page 14: Least square fit of a line/polynomial to input/output data · 2015. 7. 16. · Least square t of a line/polynomial to input/output data Prashant Dave Chemical Engg., Indian Institute](https://reader035.vdocuments.net/reader035/viewer/2022071514/613585b80ad5d20676476e54/html5/thumbnails/14.jpg)
Outline Scilab Least squares
Noise generation
Generate a vector of length n from a normal distribution withmean a and standard deviation b.
1 rand(’seed’,10): get repeatable random numbers by initializingseed.
2 rand(’normal’): generate from a normal distribution.
3 rand(x): generate a vector of same length as x.
4 a+b*rand(x): generate with mean a and standard deviation b.
Prashant Dave Least square fit
![Page 15: Least square fit of a line/polynomial to input/output data · 2015. 7. 16. · Least square t of a line/polynomial to input/output data Prashant Dave Chemical Engg., Indian Institute](https://reader035.vdocuments.net/reader035/viewer/2022071514/613585b80ad5d20676476e54/html5/thumbnails/15.jpg)
Outline Scilab Least squares
Noise generation
Generate a vector of length n from a normal distribution withmean a and standard deviation b.
1 rand(’seed’,10): get repeatable random numbers by initializingseed.
2 rand(’normal’): generate from a normal distribution.
3 rand(x): generate a vector of same length as x.
4 a+b*rand(x): generate with mean a and standard deviation b.
Prashant Dave Least square fit
![Page 16: Least square fit of a line/polynomial to input/output data · 2015. 7. 16. · Least square t of a line/polynomial to input/output data Prashant Dave Chemical Engg., Indian Institute](https://reader035.vdocuments.net/reader035/viewer/2022071514/613585b80ad5d20676476e54/html5/thumbnails/16.jpg)
Outline Scilab Least squares
Noise generation
Generate a vector of length n from a normal distribution withmean a and standard deviation b.
1 rand(’seed’,10): get repeatable random numbers by initializingseed.
2 rand(’normal’): generate from a normal distribution.
3 rand(x): generate a vector of same length as x.
4 a+b*rand(x): generate with mean a and standard deviation b.
Prashant Dave Least square fit
![Page 17: Least square fit of a line/polynomial to input/output data · 2015. 7. 16. · Least square t of a line/polynomial to input/output data Prashant Dave Chemical Engg., Indian Institute](https://reader035.vdocuments.net/reader035/viewer/2022071514/613585b80ad5d20676476e54/html5/thumbnails/17.jpg)
Outline Scilab Least squares
Noise generation
Generate a vector of length n from a normal distribution withmean a and standard deviation b.
1 rand(’seed’,10): get repeatable random numbers by initializingseed.
2 rand(’normal’): generate from a normal distribution.
3 rand(x): generate a vector of same length as x.
4 a+b*rand(x): generate with mean a and standard deviation b.
Prashant Dave Least square fit
![Page 18: Least square fit of a line/polynomial to input/output data · 2015. 7. 16. · Least square t of a line/polynomial to input/output data Prashant Dave Chemical Engg., Indian Institute](https://reader035.vdocuments.net/reader035/viewer/2022071514/613585b80ad5d20676476e54/html5/thumbnails/18.jpg)
Outline Scilab Least squares
Noise generation
Generate a vector of length n from a normal distribution withmean a and standard deviation b.
1 rand(’seed’,10): get repeatable random numbers by initializingseed.
2 rand(’normal’): generate from a normal distribution.
3 rand(x): generate a vector of same length as x.
4 a+b*rand(x): generate with mean a and standard deviation b.
Prashant Dave Least square fit
![Page 19: Least square fit of a line/polynomial to input/output data · 2015. 7. 16. · Least square t of a line/polynomial to input/output data Prashant Dave Chemical Engg., Indian Institute](https://reader035.vdocuments.net/reader035/viewer/2022071514/613585b80ad5d20676476e54/html5/thumbnails/19.jpg)
Outline Scilab Least squares
Plots for example
true line
noisy datafitted line
0
5
10
15
20
25
30
0 1 2 3 4 5 6 7 8 9 10x
y
Prashant Dave Least square fit
![Page 20: Least square fit of a line/polynomial to input/output data · 2015. 7. 16. · Least square t of a line/polynomial to input/output data Prashant Dave Chemical Engg., Indian Institute](https://reader035.vdocuments.net/reader035/viewer/2022071514/613585b80ad5d20676476e54/html5/thumbnails/20.jpg)
Outline Scilab Least squares
Higher order polynomial least-square fit
Suppose we expect yi satisfies the following equation:
yi = a2x2i + a1xi + a0
Points (xi , yi ) are sitting on a parabola.Problem (more generally):Given n samples of (xi , yi ) pairs and some choice of degree d .
yi = adxdi + ad−1x
d−1i + ... + a1xi + a0
Find constants ad , . . . a1 and a0 such that the ‘total square error’
n∑i=1
(adxdi + ad−1x
d−1i + · · · + a1xi + a0 − yi )
2 (3)
is least.
Prashant Dave Least square fit
![Page 21: Least square fit of a line/polynomial to input/output data · 2015. 7. 16. · Least square t of a line/polynomial to input/output data Prashant Dave Chemical Engg., Indian Institute](https://reader035.vdocuments.net/reader035/viewer/2022071514/613585b80ad5d20676476e54/html5/thumbnails/21.jpg)
Outline Scilab Least squares
Higher order polynomial least-square fit
Suppose we expect yi satisfies the following equation:
yi = a2x2i + a1xi + a0
Points (xi , yi ) are sitting on a parabola.
Problem (more generally):Given n samples of (xi , yi ) pairs and some choice of degree d .
yi = adxdi + ad−1x
d−1i + ... + a1xi + a0
Find constants ad , . . . a1 and a0 such that the ‘total square error’
n∑i=1
(adxdi + ad−1x
d−1i + · · · + a1xi + a0 − yi )
2 (3)
is least.
Prashant Dave Least square fit
![Page 22: Least square fit of a line/polynomial to input/output data · 2015. 7. 16. · Least square t of a line/polynomial to input/output data Prashant Dave Chemical Engg., Indian Institute](https://reader035.vdocuments.net/reader035/viewer/2022071514/613585b80ad5d20676476e54/html5/thumbnails/22.jpg)
Outline Scilab Least squares
Higher order polynomial least-square fit
Suppose we expect yi satisfies the following equation:
yi = a2x2i + a1xi + a0
Points (xi , yi ) are sitting on a parabola.Problem (more generally):
Given n samples of (xi , yi ) pairs and some choice of degree d .
yi = adxdi + ad−1x
d−1i + ... + a1xi + a0
Find constants ad , . . . a1 and a0 such that the ‘total square error’
n∑i=1
(adxdi + ad−1x
d−1i + · · · + a1xi + a0 − yi )
2 (3)
is least.
Prashant Dave Least square fit
![Page 23: Least square fit of a line/polynomial to input/output data · 2015. 7. 16. · Least square t of a line/polynomial to input/output data Prashant Dave Chemical Engg., Indian Institute](https://reader035.vdocuments.net/reader035/viewer/2022071514/613585b80ad5d20676476e54/html5/thumbnails/23.jpg)
Outline Scilab Least squares
Higher order polynomial least-square fit
Suppose we expect yi satisfies the following equation:
yi = a2x2i + a1xi + a0
Points (xi , yi ) are sitting on a parabola.Problem (more generally):Given n samples of (xi , yi ) pairs and some choice of degree d .
yi = adxdi + ad−1x
d−1i + ... + a1xi + a0
Find constants ad , . . . a1 and a0 such that the ‘total square error’
n∑i=1
(adxdi + ad−1x
d−1i + · · · + a1xi + a0 − yi )
2 (3)
is least.
Prashant Dave Least square fit
![Page 24: Least square fit of a line/polynomial to input/output data · 2015. 7. 16. · Least square t of a line/polynomial to input/output data Prashant Dave Chemical Engg., Indian Institute](https://reader035.vdocuments.net/reader035/viewer/2022071514/613585b80ad5d20676476e54/html5/thumbnails/24.jpg)
Outline Scilab Least squares
Higher order polynomial least-square fit
Suppose we expect yi satisfies the following equation:
yi = a2x2i + a1xi + a0
Points (xi , yi ) are sitting on a parabola.Problem (more generally):Given n samples of (xi , yi ) pairs and some choice of degree d .
yi = adxdi + ad−1x
d−1i + ... + a1xi + a0
Find constants ad , . . . a1 and a0 such that the ‘total square error’
n∑i=1
(adxdi + ad−1x
d−1i + · · · + a1xi + a0 − yi )
2 (3)
is least.Prashant Dave Least square fit
![Page 25: Least square fit of a line/polynomial to input/output data · 2015. 7. 16. · Least square t of a line/polynomial to input/output data Prashant Dave Chemical Engg., Indian Institute](https://reader035.vdocuments.net/reader035/viewer/2022071514/613585b80ad5d20676476e54/html5/thumbnails/25.jpg)
Outline Scilab Least squares
Still a linear regression problem
The unknowns ai enter the problem linearly.
(i.e. ai ’s are not getting squared, or multiplied to each other.)[slopes, intercept] = reglin(X,y)where X = [x ; x2]: a matrix with two regressors (one in each row)y : a row vector with same number of columns as X .slopes: the coefficients a1,a2intercept: the coefficient a0sig : standard deviation of the residual.
Prashant Dave Least square fit
![Page 26: Least square fit of a line/polynomial to input/output data · 2015. 7. 16. · Least square t of a line/polynomial to input/output data Prashant Dave Chemical Engg., Indian Institute](https://reader035.vdocuments.net/reader035/viewer/2022071514/613585b80ad5d20676476e54/html5/thumbnails/26.jpg)
Outline Scilab Least squares
Still a linear regression problem
The unknowns ai enter the problem linearly.(i.e. ai ’s are not getting squared, or multiplied to each other.)[slopes, intercept] = reglin(X,y)where X = [x ; x2]: a matrix with two regressors (one in each row)y : a row vector with same number of columns as X .slopes: the coefficients a1,a2intercept: the coefficient a0sig : standard deviation of the residual.
Prashant Dave Least square fit
![Page 27: Least square fit of a line/polynomial to input/output data · 2015. 7. 16. · Least square t of a line/polynomial to input/output data Prashant Dave Chemical Engg., Indian Institute](https://reader035.vdocuments.net/reader035/viewer/2022071514/613585b80ad5d20676476e54/html5/thumbnails/27.jpg)
Outline Scilab Least squares
Second order fit example
true data
noisy linefitted line
0
20
40
60
80
100
120
140
0 1 2 3 4 5 6 7 8 9 10x
y
Prashant Dave Least square fit
![Page 28: Least square fit of a line/polynomial to input/output data · 2015. 7. 16. · Least square t of a line/polynomial to input/output data Prashant Dave Chemical Engg., Indian Institute](https://reader035.vdocuments.net/reader035/viewer/2022071514/613585b80ad5d20676476e54/html5/thumbnails/28.jpg)
Outline Scilab Least squares
More than one independent variables
Suppose y depends on independent variables x1, x2, etc.
yi = a0 + a1x1i + a2x2i + ... + apxpi
A multiple linear regression problem (coefficients ai still appearlinearly)
[slopes,intercept]=reglin(X,y)where X and y are matrix/vector with same number of columns.(but X has many rows.)Components in slopes = number of rows of X(number of independent variables.)
Prashant Dave Least square fit
![Page 29: Least square fit of a line/polynomial to input/output data · 2015. 7. 16. · Least square t of a line/polynomial to input/output data Prashant Dave Chemical Engg., Indian Institute](https://reader035.vdocuments.net/reader035/viewer/2022071514/613585b80ad5d20676476e54/html5/thumbnails/29.jpg)
Outline Scilab Least squares
More than one independent variables
Suppose y depends on independent variables x1, x2, etc.
yi = a0 + a1x1i + a2x2i + ... + apxpi
A multiple linear regression problem (coefficients ai still appearlinearly)[slopes,intercept]=reglin(X,y)where X and y are matrix/vector with same number of columns.
(but X has many rows.)Components in slopes = number of rows of X(number of independent variables.)
Prashant Dave Least square fit
![Page 30: Least square fit of a line/polynomial to input/output data · 2015. 7. 16. · Least square t of a line/polynomial to input/output data Prashant Dave Chemical Engg., Indian Institute](https://reader035.vdocuments.net/reader035/viewer/2022071514/613585b80ad5d20676476e54/html5/thumbnails/30.jpg)
Outline Scilab Least squares
More than one independent variables
Suppose y depends on independent variables x1, x2, etc.
yi = a0 + a1x1i + a2x2i + ... + apxpi
A multiple linear regression problem (coefficients ai still appearlinearly)[slopes,intercept]=reglin(X,y)where X and y are matrix/vector with same number of columns.(but X has many rows.)Components in slopes = number of rows of X(number of independent variables.)
Prashant Dave Least square fit
![Page 31: Least square fit of a line/polynomial to input/output data · 2015. 7. 16. · Least square t of a line/polynomial to input/output data Prashant Dave Chemical Engg., Indian Institute](https://reader035.vdocuments.net/reader035/viewer/2022071514/613585b80ad5d20676476e54/html5/thumbnails/31.jpg)
Outline Scilab Least squares
Nonlinear Least Squares
The parameters to be estimated appear non-linearly in the model:y = f (x)Example, yi = a/(b + xi )
Want to choose parameters so as to minimize∑ni=1(yi − f (xi ))2.
Analytical solution usually not available.
Use a numerical optimization technique.
Scilab functions: lsqrsolve, leastsq (front end to optimfunction)
Prashant Dave Least square fit
![Page 32: Least square fit of a line/polynomial to input/output data · 2015. 7. 16. · Least square t of a line/polynomial to input/output data Prashant Dave Chemical Engg., Indian Institute](https://reader035.vdocuments.net/reader035/viewer/2022071514/613585b80ad5d20676476e54/html5/thumbnails/32.jpg)
Outline Scilab Least squares
Nonlinear Least Squares
The parameters to be estimated appear non-linearly in the model:y = f (x)Example, yi = a/(b + xi )
Want to choose parameters so as to minimize∑ni=1(yi − f (xi ))2.
Analytical solution usually not available.
Use a numerical optimization technique.
Scilab functions: lsqrsolve, leastsq (front end to optimfunction)
Prashant Dave Least square fit
![Page 33: Least square fit of a line/polynomial to input/output data · 2015. 7. 16. · Least square t of a line/polynomial to input/output data Prashant Dave Chemical Engg., Indian Institute](https://reader035.vdocuments.net/reader035/viewer/2022071514/613585b80ad5d20676476e54/html5/thumbnails/33.jpg)
Outline Scilab Least squares
Thank You
Prashant Dave Least square fit