![Page 1: Advanced Engineering Statistics - Section 5Advanced Engineering Statistics - Section 5 - Jay Liu Dept. Chemical Engineering PKNU . Least squares regression • What we will cover 2012-05-16](https://reader030.vdocuments.net/reader030/viewer/2022041003/5ea4fbcb66e9007d72200585/html5/thumbnails/1.jpg)
Advanced Engineering Statistics
- Section 5 -
Jay Liu
Dept. Chemical Engineering
PKNU
![Page 2: Advanced Engineering Statistics - Section 5Advanced Engineering Statistics - Section 5 - Jay Liu Dept. Chemical Engineering PKNU . Least squares regression • What we will cover 2012-05-16](https://reader030.vdocuments.net/reader030/viewer/2022041003/5ea4fbcb66e9007d72200585/html5/thumbnails/2.jpg)
Least squares regression
• What we will cover
2012-05-16 1 Adv. Eng. Stat., Jay Liu©
Box, G.E.P., Use and abuse of regression, Technometrics, 8 (4), 625-629, 1966
![Page 3: Advanced Engineering Statistics - Section 5Advanced Engineering Statistics - Section 5 - Jay Liu Dept. Chemical Engineering PKNU . Least squares regression • What we will cover 2012-05-16](https://reader030.vdocuments.net/reader030/viewer/2022041003/5ea4fbcb66e9007d72200585/html5/thumbnails/3.jpg)
[FYI]Least squares vs. interpolation
Given the data, there are two choices when we want to know the value
of y at x = (x1 + x2)/2
least squares? or interpolation?
Interpolation is recommended when data are subject to negligible
experimental error (or noise)
Ex. In using steam tables
Otherwise, least squares is recommended.
2012-05-16 Adv. Eng. Stat., Jay Liu© 2
x
y
x1 x2
x y
… …
… …
x1 y1
x2 y2
… …
![Page 4: Advanced Engineering Statistics - Section 5Advanced Engineering Statistics - Section 5 - Jay Liu Dept. Chemical Engineering PKNU . Least squares regression • What we will cover 2012-05-16](https://reader030.vdocuments.net/reader030/viewer/2022041003/5ea4fbcb66e9007d72200585/html5/thumbnails/4.jpg)
Least squares - usage examples
Quantify relationship between 2 variables (or 2 sets of variables):
Manager: How does yield from the lactic acid batch fermentation relate to
the purity of sucrose?
Engineer: The yield can be predicted from sucrose purity with an error of
plus/minus 8%
Manager: And how about the relationship between yield and glucose
purity?
Engineer: Over the range of our historical data, there is no discernible
relationship.
2012-05-16 Adv. Eng. Stat., Jay Liu© 3
![Page 5: Advanced Engineering Statistics - Section 5Advanced Engineering Statistics - Section 5 - Jay Liu Dept. Chemical Engineering PKNU . Least squares regression • What we will cover 2012-05-16](https://reader030.vdocuments.net/reader030/viewer/2022041003/5ea4fbcb66e9007d72200585/html5/thumbnails/5.jpg)
Least squares - usage examples
Two general applications
Predictive modeling – usually when an exact model form is unknown.
Modeling data trends in order to predict future y values
Simulation – usually when parameters in the model are unknown.
Getting parameter values in the known model form (e.g., calculate
activation energy from reaction data)
Terminology
y : response variables, output variables, dependent variables, …
x : input variables, regressor variables, independent variables, …
2012-05-16 Adv. Eng. Stat., Jay Liu© 4
![Page 6: Advanced Engineering Statistics - Section 5Advanced Engineering Statistics - Section 5 - Jay Liu Dept. Chemical Engineering PKNU . Least squares regression • What we will cover 2012-05-16](https://reader030.vdocuments.net/reader030/viewer/2022041003/5ea4fbcb66e9007d72200585/html5/thumbnails/6.jpg)
Review: covariance
Consider measurements from a gas cylinder: temperature (K) and
pressure (kPa).
Ideal gas law applies under moderate condition: pV = nRT
Fixed volume, V = 20 × 10−3m3 = 20 L
Moles of gas, n = 14.1 mols of chlorine gas, (1 kg gas)
Gas constant, R = 8.314 J/(mol.K)
Simplify the ideal gas law to: p = b1T, where
2012-05-16 Adv. Eng. Stat., Jay Liu© 5
1
nR
Vb
![Page 7: Advanced Engineering Statistics - Section 5Advanced Engineering Statistics - Section 5 - Jay Liu Dept. Chemical Engineering PKNU . Least squares regression • What we will cover 2012-05-16](https://reader030.vdocuments.net/reader030/viewer/2022041003/5ea4fbcb66e9007d72200585/html5/thumbnails/7.jpg)
Review: covariance (Cont.)
2012-05-16 Adv. Eng. Stat., Jay Liu© 6
![Page 8: Advanced Engineering Statistics - Section 5Advanced Engineering Statistics - Section 5 - Jay Liu Dept. Chemical Engineering PKNU . Least squares regression • What we will cover 2012-05-16](https://reader030.vdocuments.net/reader030/viewer/2022041003/5ea4fbcb66e9007d72200585/html5/thumbnails/8.jpg)
Review: covariance (Cont.)
Formal definition:
1. Calculate deviation variables:
Subtracting off mean centers the vector at zero.
2. Multiply the centered values:
16740 10080 5400 1440 180 60 1620 5700 10920 15660
3. Calculate the expected value (mean): 6780
4. Covariance has units: [K∙kPa]
c.f) Covariance between temperature and humidity is 202 [K∙%]
※ Covariance with itself is the variance:
2012-05-16 Adv. Eng. Stat., Jay Liu© 7
cov( , ) where ( )x y E x x y y E z z
and T T p p
T T p p
cov( , ) ( )x x V x E x x x x
T
centered centeredor T p
![Page 9: Advanced Engineering Statistics - Section 5Advanced Engineering Statistics - Section 5 - Jay Liu Dept. Chemical Engineering PKNU . Least squares regression • What we will cover 2012-05-16](https://reader030.vdocuments.net/reader030/viewer/2022041003/5ea4fbcb66e9007d72200585/html5/thumbnails/9.jpg)
Review: correlation
Q: Which one (pressure or temperature) has stronger relationship with
temperature?
Covariance depends on units: e.g. different covariance for grams vs
kilograms
Correlation removes the scaling effect:
Divides by the units of x and y: dimensionless result
Gas cylinder example:
corr(temperature, pressure) = 0.997
corr(temperature, humidity) = 0.380
2012-05-16 Adv. Eng. Stat., Jay Liu© 8
cov( , )( , )
x y x y
E x x y yx ycorr x y
1 ( , ) 1xycorr x y
![Page 10: Advanced Engineering Statistics - Section 5Advanced Engineering Statistics - Section 5 - Jay Liu Dept. Chemical Engineering PKNU . Least squares regression • What we will cover 2012-05-16](https://reader030.vdocuments.net/reader030/viewer/2022041003/5ea4fbcb66e9007d72200585/html5/thumbnails/10.jpg)
Review: correlation (cont.)
2012-05-16 Adv. Eng. Stat., Jay Liu© 9
Want to find a relationship y = f(x) other than the above?
![Page 11: Advanced Engineering Statistics - Section 5Advanced Engineering Statistics - Section 5 - Jay Liu Dept. Chemical Engineering PKNU . Least squares regression • What we will cover 2012-05-16](https://reader030.vdocuments.net/reader030/viewer/2022041003/5ea4fbcb66e9007d72200585/html5/thumbnails/11.jpg)
Review: correlation (cont.)
Remember!
2012-05-16 Adv. Eng. Stat., Jay Liu© 10
?
![Page 12: Advanced Engineering Statistics - Section 5Advanced Engineering Statistics - Section 5 - Jay Liu Dept. Chemical Engineering PKNU . Least squares regression • What we will cover 2012-05-16](https://reader030.vdocuments.net/reader030/viewer/2022041003/5ea4fbcb66e9007d72200585/html5/thumbnails/12.jpg)
Least squares? Least squares regression?
Regression is the act of choosing the “best” values for the unknown
parameters in a model on the basis of a set of measured data.
Linear regression is the special case where the model is linear in the
parameters. A straight line has the form:
There are many possible ways to define the “best” fit. However, the
most commonly used measure for bestness is the sum of squared
residuals.
Least sum of squares of errors least squares in short.
Important: error is from y, not from x.
2012-05-16 Adv. Eng. Stat., Jay Liu© 11
0 1 ( )y a a x e
![Page 13: Advanced Engineering Statistics - Section 5Advanced Engineering Statistics - Section 5 - Jay Liu Dept. Chemical Engineering PKNU . Least squares regression • What we will cover 2012-05-16](https://reader030.vdocuments.net/reader030/viewer/2022041003/5ea4fbcb66e9007d72200585/html5/thumbnails/13.jpg)
[FYI] why minimize the sum of squares ?
The least squares model:
has the lowest possible variance for a0 and a1 when certain assumptions are
met (more later)
computationally tractable by hand
easy to prove various mathematical properties
intuitive: penalize deviations quadratically
Other forms: multiple solutions, unstable, high variance solutions,
mathematical proofs are difficult
2012-05-16 Adv. Eng. Stat., Jay Liu© 12
![Page 14: Advanced Engineering Statistics - Section 5Advanced Engineering Statistics - Section 5 - Jay Liu Dept. Chemical Engineering PKNU . Least squares regression • What we will cover 2012-05-16](https://reader030.vdocuments.net/reader030/viewer/2022041003/5ea4fbcb66e9007d72200585/html5/thumbnails/14.jpg)
Least squares (regression)
It is the basis for :
DOE (Design of Experiments)
Latent variable methods
We consider only 2 (sets of) variables : x and y (or x’s and y)
Simple least squares
Multiple least squares
Generalized least squares
2012-05-16 Adv. Eng. Stat., Jay Liu© 13
![Page 15: Advanced Engineering Statistics - Section 5Advanced Engineering Statistics - Section 5 - Jay Liu Dept. Chemical Engineering PKNU . Least squares regression • What we will cover 2012-05-16](https://reader030.vdocuments.net/reader030/viewer/2022041003/5ea4fbcb66e9007d72200585/html5/thumbnails/15.jpg)
Simple least squares
Wind tunnel example
How can we find the best line that describe the following data?
2012-05-16 Adv. Eng. Stat., Jay Liu© 14
Data from wind tunnel experiments: Drag force (F) at various wind velocities
![Page 16: Advanced Engineering Statistics - Section 5Advanced Engineering Statistics - Section 5 - Jay Liu Dept. Chemical Engineering PKNU . Least squares regression • What we will cover 2012-05-16](https://reader030.vdocuments.net/reader030/viewer/2022041003/5ea4fbcb66e9007d72200585/html5/thumbnails/16.jpg)
Wind tunnel example (cont.)
From the plot, a linear line seems adequate.
y = a0 + a1x
At a data point (xi, yi), error between the line
and the point is: (see the figure on the right)
ei = yi – = yi – a0 – a1xi
Earlier, least squares means least sum of
squares of errors. For all data points, sum
of squares of errors is:
We need to find model parameters a0 and a1 that minimize Sr.
“Least squares”
2012-05-16 Adv. Eng. Stat., Jay Liu© 15
ˆiy
![Page 17: Advanced Engineering Statistics - Section 5Advanced Engineering Statistics - Section 5 - Jay Liu Dept. Chemical Engineering PKNU . Least squares regression • What we will cover 2012-05-16](https://reader030.vdocuments.net/reader030/viewer/2022041003/5ea4fbcb66e9007d72200585/html5/thumbnails/17.jpg)
Wind tunnel example (cont.)
How to find model parameters?
Take a look at Sr.
Sr is a parabolic function w.r.t ao and a1
and sign of are plus.
Sr becomes minimum where
2012-05-16 Adv. Eng. Stat., Jay Liu© 16
a0
a1
2 2
1 and oa a
0 1
0 & 0.r rS S
a a
Rearranging and
solving for a0 and a1
![Page 18: Advanced Engineering Statistics - Section 5Advanced Engineering Statistics - Section 5 - Jay Liu Dept. Chemical Engineering PKNU . Least squares regression • What we will cover 2012-05-16](https://reader030.vdocuments.net/reader030/viewer/2022041003/5ea4fbcb66e9007d72200585/html5/thumbnails/18.jpg)
Wind tunnel example (cont.)
Calculations
2012-05-16 Adv. Eng. Stat., Jay Liu© 17
![Page 19: Advanced Engineering Statistics - Section 5Advanced Engineering Statistics - Section 5 - Jay Liu Dept. Chemical Engineering PKNU . Least squares regression • What we will cover 2012-05-16](https://reader030.vdocuments.net/reader030/viewer/2022041003/5ea4fbcb66e9007d72200585/html5/thumbnails/19.jpg)
Wind tunnel example (cont.)
Calculations
This is called simple least squares.
2012-05-16 Adv. Eng. Stat., Jay Liu© 18
![Page 20: Advanced Engineering Statistics - Section 5Advanced Engineering Statistics - Section 5 - Jay Liu Dept. Chemical Engineering PKNU . Least squares regression • What we will cover 2012-05-16](https://reader030.vdocuments.net/reader030/viewer/2022041003/5ea4fbcb66e9007d72200585/html5/thumbnails/20.jpg)
Wind tunnel example (cont.)
Results
Is this OK with you?
2012-05-16 Adv. Eng. Stat., Jay Liu© 19
![Page 21: Advanced Engineering Statistics - Section 5Advanced Engineering Statistics - Section 5 - Jay Liu Dept. Chemical Engineering PKNU . Least squares regression • What we will cover 2012-05-16](https://reader030.vdocuments.net/reader030/viewer/2022041003/5ea4fbcb66e9007d72200585/html5/thumbnails/21.jpg)
General modeling procedure
2012-05-16 Adv. Eng. Stat., Jay Liu© 20
Define modeling objective
Variable selection Identify the response variables (i.e., y variables), and the regressor variables (i.e., x variables) that are to be
considered
Design of experiment Design an experiment and use it to generate the data
that will be used to fit the model
Define the model Choose an appropriate form for the model
Fit the model Estimate values for the parameters in the model
Does the model fit?
N
Y
Use the model
Statistical tools + prior knowledge
![Page 22: Advanced Engineering Statistics - Section 5Advanced Engineering Statistics - Section 5 - Jay Liu Dept. Chemical Engineering PKNU . Least squares regression • What we will cover 2012-05-16](https://reader030.vdocuments.net/reader030/viewer/2022041003/5ea4fbcb66e9007d72200585/html5/thumbnails/22.jpg)
Simple least squares
Summary
Model form: y = a0 + a1x + e
becomes minimizes where
Rearranging and solving for a0 and a1
2012-05-16 Adv. Eng. Stat., Jay Liu© 21
0 1
0 & 0.r rS S
a a
![Page 23: Advanced Engineering Statistics - Section 5Advanced Engineering Statistics - Section 5 - Jay Liu Dept. Chemical Engineering PKNU . Least squares regression • What we will cover 2012-05-16](https://reader030.vdocuments.net/reader030/viewer/2022041003/5ea4fbcb66e9007d72200585/html5/thumbnails/23.jpg)
Simple least squares (cont.)
Properties
2012-05-16 Adv. Eng. Stat., Jay Liu© 22
ˆ. ., 0i ii e y y
a0 a1
![Page 24: Advanced Engineering Statistics - Section 5Advanced Engineering Statistics - Section 5 - Jay Liu Dept. Chemical Engineering PKNU . Least squares regression • What we will cover 2012-05-16](https://reader030.vdocuments.net/reader030/viewer/2022041003/5ea4fbcb66e9007d72200585/html5/thumbnails/24.jpg)
Simple least squares (cont.)
Questions
what if our model we want to find is non-linear?
Ex. Activation energy in rate constant
Linearize !
2012-05-16 Adv. Eng. Stat., Jay Liu© 23
0
ERTk k e
a1
a1 a0
a1
a0
a1
![Page 25: Advanced Engineering Statistics - Section 5Advanced Engineering Statistics - Section 5 - Jay Liu Dept. Chemical Engineering PKNU . Least squares regression • What we will cover 2012-05-16](https://reader030.vdocuments.net/reader030/viewer/2022041003/5ea4fbcb66e9007d72200585/html5/thumbnails/25.jpg)
Linearization
Want to model non-linear relationships between independent (x) and
dependent (y) variables.
1. Make a simple linear model through a suitable transformation.
y = f(x) + e y = a0 + a1x + e
2. Use previous results (simple least squares)
※Caution: nonlinear transformation also changes P.D.F of variables (and
errors)
We will discuss about this in model assessment.
2012-05-16 Adv. Eng. Stat., Jay Liu© 24
![Page 26: Advanced Engineering Statistics - Section 5Advanced Engineering Statistics - Section 5 - Jay Liu Dept. Chemical Engineering PKNU . Least squares regression • What we will cover 2012-05-16](https://reader030.vdocuments.net/reader030/viewer/2022041003/5ea4fbcb66e9007d72200585/html5/thumbnails/26.jpg)
Linearization (Cont.)
2012-05-16 Adv. Eng. Stat., Jay Liu© 25
![Page 27: Advanced Engineering Statistics - Section 5Advanced Engineering Statistics - Section 5 - Jay Liu Dept. Chemical Engineering PKNU . Least squares regression • What we will cover 2012-05-16](https://reader030.vdocuments.net/reader030/viewer/2022041003/5ea4fbcb66e9007d72200585/html5/thumbnails/27.jpg)
Polynomial regression
For quadratic form
Sum of squares
Again, Sr has a parabolic shape w.r.t a0, a1, and a2. with plus signs of
2012-05-16 Adv. Eng. Stat., Jay Liu© 26
2 2 2
0 1 2, , and .a a a
2
0 1 2
0
2
0 1 2
1
2 2
0 1 2
2
2 ( ) 0
2 ( ) 0
2 ( ) 0
ri i i
ri i i i
ri i i i
Sy a a x a x
a
Sx y a a x a x
a
Sx y a a x a x
a
![Page 28: Advanced Engineering Statistics - Section 5Advanced Engineering Statistics - Section 5 - Jay Liu Dept. Chemical Engineering PKNU . Least squares regression • What we will cover 2012-05-16](https://reader030.vdocuments.net/reader030/viewer/2022041003/5ea4fbcb66e9007d72200585/html5/thumbnails/28.jpg)
Polynomial regression (Cont.)
Rearranging the previous equations gives
the above equations can be solved easily. (three unknowns and three
equations.)
For general polynomials
From the results of two cases (y = a0 + a1x & y = a0 + a1x + a2x2)
we need to solve (m+1) linear algebraic equations for (m+1) parameters.
2012-05-16 Adv. Eng. Stat., Jay Liu© 27
0 1
0r r r
m
S S S
a a a
2
0
2 3
1
2 3 4 2
2
i i i
i i i i i
i i i i i
n x x a y
x x x a x y
x x x a x y
![Page 29: Advanced Engineering Statistics - Section 5Advanced Engineering Statistics - Section 5 - Jay Liu Dept. Chemical Engineering PKNU . Least squares regression • What we will cover 2012-05-16](https://reader030.vdocuments.net/reader030/viewer/2022041003/5ea4fbcb66e9007d72200585/html5/thumbnails/29.jpg)
Multiple least squares
Consider when there are more than two independent variables, x1, x2,
…, xm. regression plane.
For 2-D case, y = a0 + a1x1 + a2x2.
Again, Sr has a parabolic shape w.r.t a0, a1.
2012-05-16 Adv. Eng. Stat., Jay Liu© 28
exaxaxaay mm 22110
2
,22,110 )( iiir xaxaayS
0 1 1, 2 2,
0
1, 0 1 1, 2 2,
1
2, 0 1 1, 2 2,
2
2 ( ) 0
2 ( ) 0
2 ( ) 0
ri i i
ri i i i
ri i i i
Sy a a x a x
a
Sx y a a x a x
a
Sx y a a x a x
a
a1 a2
![Page 30: Advanced Engineering Statistics - Section 5Advanced Engineering Statistics - Section 5 - Jay Liu Dept. Chemical Engineering PKNU . Least squares regression • What we will cover 2012-05-16](https://reader030.vdocuments.net/reader030/viewer/2022041003/5ea4fbcb66e9007d72200585/html5/thumbnails/30.jpg)
Multiple least squares (Cont.)
Rearranging and solve for a0, a1 and a2 gives
For an m-dimensional plane,
Same as in general polynomials,
we need to solve (m+1) linear algebraic equations for (m+1) parameters.
2012-05-16 Adv. Eng. Stat., Jay Liu© 29
exaxaxaay mm 22110
0 1
0r r r
m
S S S
a a a
![Page 31: Advanced Engineering Statistics - Section 5Advanced Engineering Statistics - Section 5 - Jay Liu Dept. Chemical Engineering PKNU . Least squares regression • What we will cover 2012-05-16](https://reader030.vdocuments.net/reader030/viewer/2022041003/5ea4fbcb66e9007d72200585/html5/thumbnails/31.jpg)
General least squares
The following form includes all cases (simple least squares, polynomial
regression, multiple regression)
Ex. Simple and multiple least squares
polynomial regression
Same as before,
we need to solve (m+1) linear algebraic equations for (m+1) parameters.
2012-05-16 Adv. Eng. Stat., Jay Liu© 30
0 1
0r r r
m
S S S
a a a
![Page 32: Advanced Engineering Statistics - Section 5Advanced Engineering Statistics - Section 5 - Jay Liu Dept. Chemical Engineering PKNU . Least squares regression • What we will cover 2012-05-16](https://reader030.vdocuments.net/reader030/viewer/2022041003/5ea4fbcb66e9007d72200585/html5/thumbnails/32.jpg)
Quantification of errors
2012-05-16 Adv. Eng. Stat., Jay Liu© 31
2
yyS it
2
,,11,00
2
immiii
ir
zazazay
eS
Total sum of squares around the mean for the response variable, y
Sum of squares of residuals around the regression line
![Page 33: Advanced Engineering Statistics - Section 5Advanced Engineering Statistics - Section 5 - Jay Liu Dept. Chemical Engineering PKNU . Least squares regression • What we will cover 2012-05-16](https://reader030.vdocuments.net/reader030/viewer/2022041003/5ea4fbcb66e9007d72200585/html5/thumbnails/33.jpg)
Quantification of errors (Cont.)
2012-05-16 Adv. Eng. Stat., Jay Liu© 32
11
1 2
n
Syy
nS t
iy)1(
mn
SS r
xy
Standard error of predicted y (SE)
quantify appropriateness of
regression
Standard deviation of y
![Page 34: Advanced Engineering Statistics - Section 5Advanced Engineering Statistics - Section 5 - Jay Liu Dept. Chemical Engineering PKNU . Least squares regression • What we will cover 2012-05-16](https://reader030.vdocuments.net/reader030/viewer/2022041003/5ea4fbcb66e9007d72200585/html5/thumbnails/34.jpg)
Quantification of errors (Cont.)
Coefficients of determination, R2
2012-05-16 Adv. Eng. Stat., Jay Liu© 33
t
rt
S
SSR
2 The amount of variability in the data explained
by the regression model.
R2 = 1 when Sr = 0 : perfect fit (a regression curve passes through data points)
R2 = 0 when Sr = St : as bad as doing nothing
It is evident from the figures that a parabola is adequate. R2 of (b) is higher than that of (a)
![Page 35: Advanced Engineering Statistics - Section 5Advanced Engineering Statistics - Section 5 - Jay Liu Dept. Chemical Engineering PKNU . Least squares regression • What we will cover 2012-05-16](https://reader030.vdocuments.net/reader030/viewer/2022041003/5ea4fbcb66e9007d72200585/html5/thumbnails/35.jpg)
Quantification of errors (Cont.)
Warning! : R2 ≈ 1 does not guarantee that the model is adequate,
nor the model will predict new data well.
It is possible to force R2 to be one by adding as many terms as there are
observations.
Sr can be big when variance of random error is large.
(Usual assumption on error is that error is random is unpredictable)
Practice using Excel
(1) Wind tunnel example with higher polynomials
(2) Simple regression with increasing random noise
2012-05-16 Adv. Eng. Stat., Jay Liu© 34
![Page 36: Advanced Engineering Statistics - Section 5Advanced Engineering Statistics - Section 5 - Jay Liu Dept. Chemical Engineering PKNU . Least squares regression • What we will cover 2012-05-16](https://reader030.vdocuments.net/reader030/viewer/2022041003/5ea4fbcb66e9007d72200585/html5/thumbnails/36.jpg)
Confidence intervals - coefficients
Coefficients in the regression model have confidence interval.
Why? They are also statistics like & s. That is, they are numerical
quantities calculated in a sample (not entire population). They are
estimated values of parameters.
2012-05-16 Adv. Eng. Stat., Jay Liu© 35
statisticstatistic A Statistic that we want to find its confidence interval
Standard error of the statistic
Value that depends on P.D.F of the statistic & confidence level a
x
statistic A statistic
za/2
tn,a/2
xx n
xxs n
※The standard error of a statistic is the standard deviation of the sampling distribution of that statistic
![Page 37: Advanced Engineering Statistics - Section 5Advanced Engineering Statistics - Section 5 - Jay Liu Dept. Chemical Engineering PKNU . Least squares regression • What we will cover 2012-05-16](https://reader030.vdocuments.net/reader030/viewer/2022041003/5ea4fbcb66e9007d72200585/html5/thumbnails/37.jpg)
Confidence intervals – coefficients (cont.)
Matrix representation of GLS
2012-05-16 Adv. Eng. Stat., Jay Liu© 36
eZay
Z
T y
Ta
Te
m+1: number of coefficients n: number of data points
![Page 38: Advanced Engineering Statistics - Section 5Advanced Engineering Statistics - Section 5 - Jay Liu Dept. Chemical Engineering PKNU . Least squares regression • What we will cover 2012-05-16](https://reader030.vdocuments.net/reader030/viewer/2022041003/5ea4fbcb66e9007d72200585/html5/thumbnails/38.jpg)
Confidence intervals – coefficients (Cont.)
Example
Fitting quadratic polynomials to five data points
2012-05-16 Adv. Eng. Stat., Jay Liu© 37
x
y
exaxaay 2
210
eZay
5
4
3
2
1
2
1
0
0.10.11
25.05.01
0.00.01
25.05.01
0.10.11
0.2
5.0
0.0
5.0
0.1
e
e
e
e
e
a
a
a
Can you solve this problem?
Three unknowns
Five equations
![Page 39: Advanced Engineering Statistics - Section 5Advanced Engineering Statistics - Section 5 - Jay Liu Dept. Chemical Engineering PKNU . Least squares regression • What we will cover 2012-05-16](https://reader030.vdocuments.net/reader030/viewer/2022041003/5ea4fbcb66e9007d72200585/html5/thumbnails/39.jpg)
Confidence intervals – coefficients (Cont.)
Solutions
1. LU decomposition or other methods to solve L.A.E
2. Matrix inversion
computationally not efficient, but statistically useful
2012-05-16 Adv. Eng. Stat., Jay Liu© 38
eZay
ZayZayee TT
ir eS 2
Sum of squares of errors
0
a
rS yZaZZTT
Called “normal equations”
yZaZZTT "" bAx
yZaZZTT yZZZa
TT 1
![Page 40: Advanced Engineering Statistics - Section 5Advanced Engineering Statistics - Section 5 - Jay Liu Dept. Chemical Engineering PKNU . Least squares regression • What we will cover 2012-05-16](https://reader030.vdocuments.net/reader030/viewer/2022041003/5ea4fbcb66e9007d72200585/html5/thumbnails/40.jpg)
Confidence intervals – coefficients (Cont.)
Matrix inversion approach
Denote as the diagonal element of
Confidence interval of estimated coefficients
2012-05-16 Adv. Eng. Stat., Jay Liu© 39
yZZZaTT 1
1ZZ
T1
iiZ
2 1
1 ( 1), /2i n m y iix
a t S Za
( 1), /2n mt a
)1(
mn
SS r
xy
Student t statistics
Standard error of estimate
![Page 41: Advanced Engineering Statistics - Section 5Advanced Engineering Statistics - Section 5 - Jay Liu Dept. Chemical Engineering PKNU . Least squares regression • What we will cover 2012-05-16](https://reader030.vdocuments.net/reader030/viewer/2022041003/5ea4fbcb66e9007d72200585/html5/thumbnails/41.jpg)
Confidence intervals – coefficients (Cont.)
For a linear model,
C.I. for a1(slope)
C.I. for a0 (intercept)
2012-05-16 Adv. Eng. Stat., Jay Liu© 40
What if confidence intervals contain zero?
2
0 ( 1), /2 / 2
1n m y x
i
xa t S
n x xa
1 ( 1), /2 / 2
1n m y x
i
a t Sx x
a
![Page 42: Advanced Engineering Statistics - Section 5Advanced Engineering Statistics - Section 5 - Jay Liu Dept. Chemical Engineering PKNU . Least squares regression • What we will cover 2012-05-16](https://reader030.vdocuments.net/reader030/viewer/2022041003/5ea4fbcb66e9007d72200585/html5/thumbnails/42.jpg)
Confidence intervals – prediction
C.I for predicted y,
2012-05-16 Adv. Eng. Stat., Jay Liu© 41
0
2
0
( 1), /2 / 2
1ˆ
n m y xx
i
x xy t S
n x xa
ˆiy
![Page 43: Advanced Engineering Statistics - Section 5Advanced Engineering Statistics - Section 5 - Jay Liu Dept. Chemical Engineering PKNU . Least squares regression • What we will cover 2012-05-16](https://reader030.vdocuments.net/reader030/viewer/2022041003/5ea4fbcb66e9007d72200585/html5/thumbnails/43.jpg)
Model assessment
When we do not know the model form, we have to assess the model
before use it after we fit a regression model.
However, in order to assess the model and make inferences about the
parameters and predictions from the model, we will have to employ
statistics and make some assumptions about the nature of the disturbance.
Tools for model assessment
Sy/x, R2 (quantitative) ( Do not use)
Residual Plots (qualitative)
Normal probability chart (qualitative or quantitative)
Test for lack of fit (quantitative)
This is used when the dataset includes replicates. It is based on
analysis of variance (ANOVA).
2012-05-16 Adv. Eng. Stat., Jay Liu© 42
![Page 44: Advanced Engineering Statistics - Section 5Advanced Engineering Statistics - Section 5 - Jay Liu Dept. Chemical Engineering PKNU . Least squares regression • What we will cover 2012-05-16](https://reader030.vdocuments.net/reader030/viewer/2022041003/5ea4fbcb66e9007d72200585/html5/thumbnails/44.jpg)
Model assessment - assumptions
What is the most desirable errors in regression ?
Assumptions on error
Error is additive
The variance of the error is constant and is not related to values of the
response or values of the regressor variables.
There is no error associated with the values of the regressor variables.
Error is a random variable with Gaussian distribution N(0,2) (2 usually
unknown)
2012-05-16 Adv. Eng. Stat., Jay Liu© 43
unpredictable
exaay 110 exaay )( 110
![Page 45: Advanced Engineering Statistics - Section 5Advanced Engineering Statistics - Section 5 - Jay Liu Dept. Chemical Engineering PKNU . Least squares regression • What we will cover 2012-05-16](https://reader030.vdocuments.net/reader030/viewer/2022041003/5ea4fbcb66e9007d72200585/html5/thumbnails/45.jpg)
Model assessment – residual plots
Recall the assumptions on error
Error is not related to the values of response or regressor variables.
Then, assumptions will not be valid if the model is wrong.
Following residual plots will reveal this.
Residuals vs. regressor variables
Residuals vs. fitted y values ( )
Residuals vs. “lurking” variables (i.e. time or order)
These plots will show “some patterns” when a model is inadequate.
2012-05-16 Adv. Eng. Stat., Jay Liu© 44
ˆiy
![Page 46: Advanced Engineering Statistics - Section 5Advanced Engineering Statistics - Section 5 - Jay Liu Dept. Chemical Engineering PKNU . Least squares regression • What we will cover 2012-05-16](https://reader030.vdocuments.net/reader030/viewer/2022041003/5ea4fbcb66e9007d72200585/html5/thumbnails/46.jpg)
Model assessment – residual plots (con’t)
Examples
2012-05-16 Adv. Eng. Stat., Jay Liu© 45
x
e
x
e
![Page 47: Advanced Engineering Statistics - Section 5Advanced Engineering Statistics - Section 5 - Jay Liu Dept. Chemical Engineering PKNU . Least squares regression • What we will cover 2012-05-16](https://reader030.vdocuments.net/reader030/viewer/2022041003/5ea4fbcb66e9007d72200585/html5/thumbnails/47.jpg)
Model assessment – residual plots (con’t)
Examples of residual plots
2012-05-16 Adv. Eng. Stat., Jay Liu© 46
![Page 48: Advanced Engineering Statistics - Section 5Advanced Engineering Statistics - Section 5 - Jay Liu Dept. Chemical Engineering PKNU . Least squares regression • What we will cover 2012-05-16](https://reader030.vdocuments.net/reader030/viewer/2022041003/5ea4fbcb66e9007d72200585/html5/thumbnails/48.jpg)
Model assessment – normal probability plot
Recall the assumptions on error
Error is a random variable with Gaussian distribution N(0, 2 ) (2 usually
unknown)
Then, errors will fall onto a straight line (y = x) in a normal probability
plot. (especially useful when the number of data points is large)
2012-05-16 Adv. Eng. Stat., Jay Liu© 47
Normal probability plot
Alternatively, normality test
can be used.
![Page 49: Advanced Engineering Statistics - Section 5Advanced Engineering Statistics - Section 5 - Jay Liu Dept. Chemical Engineering PKNU . Least squares regression • What we will cover 2012-05-16](https://reader030.vdocuments.net/reader030/viewer/2022041003/5ea4fbcb66e9007d72200585/html5/thumbnails/49.jpg)
Model assessment – ANOVA (Test for lack of fit)
The variance breakdown
2012-05-16 Adv. Eng. Stat., Jay Liu© 48
Sr
St 2
t iS y y
2
ˆr i iS y y
2
Reˆ
g iS y y Se
Really? Prove to yourself
Ret g rS S S
![Page 50: Advanced Engineering Statistics - Section 5Advanced Engineering Statistics - Section 5 - Jay Liu Dept. Chemical Engineering PKNU . Least squares regression • What we will cover 2012-05-16](https://reader030.vdocuments.net/reader030/viewer/2022041003/5ea4fbcb66e9007d72200585/html5/thumbnails/50.jpg)
Model assessment – ANOVA (Test for lack of fit)
The variance breakdown
Ratio of SReg/Sr follows F distribution when corrected with degree of
freedom.
If regression is not meaningful, the ratio (Se/Sr) is small and St ≒ Sr.
2012-05-16 Adv. Eng. Stat., Jay Liu© 49
Ret g rS S S
Sr
St 2
t iS y y
2ˆ
r i iS y y
2
Reˆ
g iS y y SReg
![Page 51: Advanced Engineering Statistics - Section 5Advanced Engineering Statistics - Section 5 - Jay Liu Dept. Chemical Engineering PKNU . Least squares regression • What we will cover 2012-05-16](https://reader030.vdocuments.net/reader030/viewer/2022041003/5ea4fbcb66e9007d72200585/html5/thumbnails/51.jpg)
Model assessment – ANOVA (Test for lack of fit)
2012-05-16 Adv. Eng. Stat., Jay Liu© 50
Re gSrStS
a1 a0
![Page 52: Advanced Engineering Statistics - Section 5Advanced Engineering Statistics - Section 5 - Jay Liu Dept. Chemical Engineering PKNU . Least squares regression • What we will cover 2012-05-16](https://reader030.vdocuments.net/reader030/viewer/2022041003/5ea4fbcb66e9007d72200585/html5/thumbnails/52.jpg)
Model assessment – ANOVA (Test for lack of fit)
2012-05-16 Adv. Eng. Stat., Jay Liu© 51
ANOVA Table
Source of Var. Sum of Squares
Degrees of Freedom
Mean Square F0
Regression SReg p MSReg=SReg/p MSReg/ MSE
Residual error Sr n-p MSE=Sr/(n-p)
Total St n-1
Compare F0 to the critical value Fp,n-p;a
What we are doing is a test of hypothesis.
We are testing the hypothesis:
H0 : 0p0 bb
H1 : at least one parameter is not equal to zero.
![Page 53: Advanced Engineering Statistics - Section 5Advanced Engineering Statistics - Section 5 - Jay Liu Dept. Chemical Engineering PKNU . Least squares regression • What we will cover 2012-05-16](https://reader030.vdocuments.net/reader030/viewer/2022041003/5ea4fbcb66e9007d72200585/html5/thumbnails/53.jpg)
[FYI]Meaning of a p-value in hypothesis test
A measure of how much evidence we have against the null hypothesis.
Null hypothesis (H0) represents the hypothesis of no change or no effect.
Much research involves making a hypothesis and then collecting data to
test that hypothesis. Then researchers will collect data and measure the
consistency of this data with the null hypothesis.
A small p-value is evidence against the null hypothesis while a large p-value
means little or no evidence against the null hypothesis.
Traditionally, researchers will reject a null hypothesis if the p-value is less
than 0.05 (a = 0.05).
p-value can mean that the possibility that you can be wrong when rejecting
the null hypothesis.
2012-05-16 Adv. Eng. Stat., Jay Liu© 52
![Page 54: Advanced Engineering Statistics - Section 5Advanced Engineering Statistics - Section 5 - Jay Liu Dept. Chemical Engineering PKNU . Least squares regression • What we will cover 2012-05-16](https://reader030.vdocuments.net/reader030/viewer/2022041003/5ea4fbcb66e9007d72200585/html5/thumbnails/54.jpg)
Integer variables in the model
Integer variables 0 and 1 can represent qualitative variables.
Example: raw material from Spain, India, or Vietnam
y = a0 + a1x1 + . . . + akxk + r1d1 + r2d2 + r3d3
d1 = 1 and d2 = 0 and d3 = 0 for Spain
d1 = 0 and d2 = 1 and d3 = 0 for India
d1 = 0 and d2 = 0 and d3 = 1 for Vietnam
Often called indicator variables for this reason
2012-05-16 Adv. Eng. Stat., Jay Liu© 53
![Page 55: Advanced Engineering Statistics - Section 5Advanced Engineering Statistics - Section 5 - Jay Liu Dept. Chemical Engineering PKNU . Least squares regression • What we will cover 2012-05-16](https://reader030.vdocuments.net/reader030/viewer/2022041003/5ea4fbcb66e9007d72200585/html5/thumbnails/55.jpg)
Integer variables in the model
Example
Want to predict yield when two different impeller
used. Yield = f(temperature, impeller type)
Build two different models
(one for axial, one for radial)
Build one model using indicator variable. y = a0 + a1T + rd
y = a0 + a1T + rdi
di = 0 for axial, di = 1 for radial
2012-05-16 Adv. Eng. Stat., Jay Liu© 54
axial radial
![Page 56: Advanced Engineering Statistics - Section 5Advanced Engineering Statistics - Section 5 - Jay Liu Dept. Chemical Engineering PKNU . Least squares regression • What we will cover 2012-05-16](https://reader030.vdocuments.net/reader030/viewer/2022041003/5ea4fbcb66e9007d72200585/html5/thumbnails/56.jpg)
Leverage effect
Unusual observations influence the model parameters and our
interpretation
To avoid the leverage effect,
Remove outliers before regression (but do not delete without investigation)
Use different Sr (no longer least squares)
2012-05-16 Adv. Eng. Stat., Jay Liu© 55
x
y
x
y
Outliers have an over-proportional effect on resulting regression curves.
![Page 57: Advanced Engineering Statistics - Section 5Advanced Engineering Statistics - Section 5 - Jay Liu Dept. Chemical Engineering PKNU . Least squares regression • What we will cover 2012-05-16](https://reader030.vdocuments.net/reader030/viewer/2022041003/5ea4fbcb66e9007d72200585/html5/thumbnails/57.jpg)
Causal relation and correlation
Causal relation
Cause and effect relation
Has physical/chemical/engineering meanings
x and y are not interchangeable
Direction exists.
Correlation
(Linear) relationship between two variables
No physical/chemical/engineering meanings.
Average height of 20’s men vs. year
x and y are interchangeable
2012-05-16 Adv. Eng. Stat., Jay Liu© 56
![Page 58: Advanced Engineering Statistics - Section 5Advanced Engineering Statistics - Section 5 - Jay Liu Dept. Chemical Engineering PKNU . Least squares regression • What we will cover 2012-05-16](https://reader030.vdocuments.net/reader030/viewer/2022041003/5ea4fbcb66e9007d72200585/html5/thumbnails/58.jpg)
Advanced topics
Testing of least-squares models
2012-05-16 Adv. Eng. Stat., Jay Liu© 57
![Page 59: Advanced Engineering Statistics - Section 5Advanced Engineering Statistics - Section 5 - Jay Liu Dept. Chemical Engineering PKNU . Least squares regression • What we will cover 2012-05-16](https://reader030.vdocuments.net/reader030/viewer/2022041003/5ea4fbcb66e9007d72200585/html5/thumbnails/59.jpg)
Advanced topics - Testing of least-squares models
2012-05-16 Adv. Eng. Stat., Jay Liu© 58
![Page 60: Advanced Engineering Statistics - Section 5Advanced Engineering Statistics - Section 5 - Jay Liu Dept. Chemical Engineering PKNU . Least squares regression • What we will cover 2012-05-16](https://reader030.vdocuments.net/reader030/viewer/2022041003/5ea4fbcb66e9007d72200585/html5/thumbnails/60.jpg)
Advanced topics
Correlated x’s
MLR solution
When two or more x’s are correlated, becomes nearly singular, i.e.,
ill-conditioned.
2012-05-16 Adv. Eng. Stat., Jay Liu© 59
y Za e 2 TT
r iS e e e y Za y Za
0rS
a T TZ Z a Z y
1
T T
a Z Z Z y
1
T
Z Z
![Page 61: Advanced Engineering Statistics - Section 5Advanced Engineering Statistics - Section 5 - Jay Liu Dept. Chemical Engineering PKNU . Least squares regression • What we will cover 2012-05-16](https://reader030.vdocuments.net/reader030/viewer/2022041003/5ea4fbcb66e9007d72200585/html5/thumbnails/61.jpg)
Advanced topics – correlated x’s
High/no correlation between x1 and x2
What if very small (measurement ) noises added to x’s
What will happen to your MLR model?
2012-05-16 Adv. Eng. Stat., Jay Liu© 60
1
1.0000 0.9999
0.9999 1.0000
5000.25 4999.75
4999.75 5000.25
T
T
Z Z
Z Z 1
1.0000 0.0
0.0 1.0000
1.0 0.0
0.0 1.0
T
T
Z Z
Z Z
1
1.0001 0.9999
0.9999 1.0000
3333.34 3333.11
3333.11 3333.34
T
T
Z Z
Z Z 1
1.0000 0.0
0.0 1.0001
0.9999 0.0
0.0 0.9999
T
T
Z Z
Z Z
![Page 62: Advanced Engineering Statistics - Section 5Advanced Engineering Statistics - Section 5 - Jay Liu Dept. Chemical Engineering PKNU . Least squares regression • What we will cover 2012-05-16](https://reader030.vdocuments.net/reader030/viewer/2022041003/5ea4fbcb66e9007d72200585/html5/thumbnails/62.jpg)
Advanced topics – correlated x’s
If high correlation among x’s:
unstable solutions for a
predictions uncertain also
Geometrically speaking
2012-05-16 Adv. Eng. Stat., Jay Liu© 61
High correlation between x1 and x2
![Page 63: Advanced Engineering Statistics - Section 5Advanced Engineering Statistics - Section 5 - Jay Liu Dept. Chemical Engineering PKNU . Least squares regression • What we will cover 2012-05-16](https://reader030.vdocuments.net/reader030/viewer/2022041003/5ea4fbcb66e9007d72200585/html5/thumbnails/63.jpg)
Advanced topics – correlated x’s
Remedies?
Use selected x variables stepwise regression
Use ridge regression
Use multivariate methods (will not be covered in this lecture)
2012-05-16 Adv. Eng. Stat., Jay Liu© 62
![Page 64: Advanced Engineering Statistics - Section 5Advanced Engineering Statistics - Section 5 - Jay Liu Dept. Chemical Engineering PKNU . Least squares regression • What we will cover 2012-05-16](https://reader030.vdocuments.net/reader030/viewer/2022041003/5ea4fbcb66e9007d72200585/html5/thumbnails/64.jpg)
Advanced topics
What we want to know:
How do we select the form of the model? Which variables should be
included? Should we include transformations of the regressor variables?
….
What we want:
We would like to build the “best” regression model
We would like to include as many regressor variables as is necessary to
adequately describe the behaviour of y. At the same time, we want to keep
the model as simple as possible.
Stepwise regression start off by choosing an equation having the single best
x variables and the attempts to build up with subsequent additions of x’s one
at a time as long as these additions are worthwhile.
2012-05-16 Adv. Eng. Stat., Jay Liu© 63
![Page 65: Advanced Engineering Statistics - Section 5Advanced Engineering Statistics - Section 5 - Jay Liu Dept. Chemical Engineering PKNU . Least squares regression • What we will cover 2012-05-16](https://reader030.vdocuments.net/reader030/viewer/2022041003/5ea4fbcb66e9007d72200585/html5/thumbnails/65.jpg)
Advanced topics – stepwise regression
Procedure
1. Add a x variable to the model (the variable that is most highly correlated
with y).
2. Check to see whether or not this has significantly improved the model. One
way is to see whether or not the confidence interval for the parameter
includes zero. (of course you can use hypothesis test) If the new term or
terms are not significant, remove them from the model.
3. Find one of the remaining x variables that is highly correlated with the
residuals and repeat the procedure.
2012-05-16 Adv. Eng. Stat., Jay Liu© 64
![Page 66: Advanced Engineering Statistics - Section 5Advanced Engineering Statistics - Section 5 - Jay Liu Dept. Chemical Engineering PKNU . Least squares regression • What we will cover 2012-05-16](https://reader030.vdocuments.net/reader030/viewer/2022041003/5ea4fbcb66e9007d72200585/html5/thumbnails/66.jpg)
Advanced topics
Ridge regression (Hoerl, 1962; Hoerl & Kennard, 1970a,b)
A modified regression method specifically for ill-conditioned datasets that
allows all variables to be kept in the model.
This is possible by adding additional information to the problem to remove
the ill-conditioning.
The objective function to minimize:
Least squares estimates have no bias but large variance, while ridge
regression estimates have small bias and small variance.
2012-05-16 Adv. Eng. Stat., Jay Liu© 65
T Tk y Xb y Xb b b
![Page 67: Advanced Engineering Statistics - Section 5Advanced Engineering Statistics - Section 5 - Jay Liu Dept. Chemical Engineering PKNU . Least squares regression • What we will cover 2012-05-16](https://reader030.vdocuments.net/reader030/viewer/2022041003/5ea4fbcb66e9007d72200585/html5/thumbnails/67.jpg)
Advanced topics – ridge regression
Procedure
1. Mean center and scale all x’s to unit variance
2. Rewrite the model as:
2012-05-16 Adv. Eng. Stat., Jay Liu© 66
i
ii
xs
x xf
1
1
11
1
p
p
p
x p x
x x
p p
a s a ss s
b b
1
x xx xy y
y f f
Y Fb ε
![Page 68: Advanced Engineering Statistics - Section 5Advanced Engineering Statistics - Section 5 - Jay Liu Dept. Chemical Engineering PKNU . Least squares regression • What we will cover 2012-05-16](https://reader030.vdocuments.net/reader030/viewer/2022041003/5ea4fbcb66e9007d72200585/html5/thumbnails/68.jpg)
Advanced topics – ridge regression
3. The objective function to use for the optimization is to minimize
Therefore,
4. Solve the optimization problem in Step 3 for several values of k between 0
and 1 and choose that value of k at which the estimates of b see to stabilize.
Otherwise, choose k by validation on new data.
2012-05-16 Adv. Eng. Stat., Jay Liu© 67
T
J k Y Fb Y Fb b b
1
* T Tk
b F FZ I F Y
![Page 69: Advanced Engineering Statistics - Section 5Advanced Engineering Statistics - Section 5 - Jay Liu Dept. Chemical Engineering PKNU . Least squares regression • What we will cover 2012-05-16](https://reader030.vdocuments.net/reader030/viewer/2022041003/5ea4fbcb66e9007d72200585/html5/thumbnails/69.jpg)
Advanced topics
Non-linear regression
General form of a non-linear regression model
In a linear model, . In a non-linear model, f() woulds have any
form. E.g.,
Remember that nonlinear transformation also changes P.D.F of variables
(and errors)? What does this mean?
2012-05-16 Adv. Eng. Stat., Jay Liu© 68
( , )y f x a
( , ) Tf x a x a
b
b
x1
ey
2
x1
![Page 70: Advanced Engineering Statistics - Section 5Advanced Engineering Statistics - Section 5 - Jay Liu Dept. Chemical Engineering PKNU . Least squares regression • What we will cover 2012-05-16](https://reader030.vdocuments.net/reader030/viewer/2022041003/5ea4fbcb66e9007d72200585/html5/thumbnails/70.jpg)
Advanced topics – non-linear regression
The approach is exactly the same as for linear models
We use the same objective function:
All we need is to minimize S over a. but how?
2012-05-16 Adv. Eng. Stat., Jay Liu© 69
2
1
2
1
ˆ( )
( , )
r
n
i i
i
n
i i
i
S
y y
y f
ε ε
x a
![Page 71: Advanced Engineering Statistics - Section 5Advanced Engineering Statistics - Section 5 - Jay Liu Dept. Chemical Engineering PKNU . Least squares regression • What we will cover 2012-05-16](https://reader030.vdocuments.net/reader030/viewer/2022041003/5ea4fbcb66e9007d72200585/html5/thumbnails/71.jpg)
Advanced topics – non-linear regression
The big difference between linear and nonlinear regression is that in
general, the optimization problem for a nonlinear model does not have
an exact analytical solution.
Therefore, we have to use a numerical optimization algorithm such as:
Gauss-Newton
Steepest Descent
Conjugated Gradients
Any other optimization algorithm
2012-05-16 Adv. Eng. Stat., Jay Liu© 70
![Page 72: Advanced Engineering Statistics - Section 5Advanced Engineering Statistics - Section 5 - Jay Liu Dept. Chemical Engineering PKNU . Least squares regression • What we will cover 2012-05-16](https://reader030.vdocuments.net/reader030/viewer/2022041003/5ea4fbcb66e9007d72200585/html5/thumbnails/72.jpg)
Advanced topics – non-linear regression
When using an optimization algorithm to solve nonlinear regression
problems, one needs to be able to specify:
1. an expectation function (i.e. the form of the model)
2. Data
3. starting guesses for a
4. stopping criteria
5. possibly other “tuning” parameters associated with the optimization
algorithm
2012-05-16 Adv. Eng. Stat., Jay Liu© 71
![Page 73: Advanced Engineering Statistics - Section 5Advanced Engineering Statistics - Section 5 - Jay Liu Dept. Chemical Engineering PKNU . Least squares regression • What we will cover 2012-05-16](https://reader030.vdocuments.net/reader030/viewer/2022041003/5ea4fbcb66e9007d72200585/html5/thumbnails/73.jpg)
Advanced topics – non-linear regression
Problems with Numerical Optimization
Failure to converge
Finding only a local minimum and not the global minimum
Requires good starting guesses for the parameters
Can be sensitive to the choice of convergence criteria and other “tuning
parameters” of the algorithm
Sometimes requires specification of the derivatives of the model with
respect the the parameters.
2012-05-16 Adv. Eng. Stat., Jay Liu© 72