multivariate linear models for regression and classification outline: 1) multivariate linear...
TRANSCRIPT
![Page 1: Multivariate linear models for regression and classification Outline: 1) multivariate linear regression 2) linear classification (perceptron) 3) logistic](https://reader036.vdocuments.net/reader036/viewer/2022082505/56649d1e5503460f949f1d2c/html5/thumbnails/1.jpg)
Multivariate linear models for regression and classification
Outline:1) multivariate linear regression2) linear classification (perceptron)3) logistic regression
![Page 2: Multivariate linear models for regression and classification Outline: 1) multivariate linear regression 2) linear classification (perceptron) 3) logistic](https://reader036.vdocuments.net/reader036/viewer/2022082505/56649d1e5503460f949f1d2c/html5/thumbnails/2.jpg)
Logistic Regression(lecture 9 on amlbook.com)
![Page 3: Multivariate linear models for regression and classification Outline: 1) multivariate linear regression 2) linear classification (perceptron) 3) logistic](https://reader036.vdocuments.net/reader036/viewer/2022082505/56649d1e5503460f949f1d2c/html5/thumbnails/3.jpg)
Neuron analogy
Dot product wTx is a way of combining attributes into a scalar signal s. How signal is used defines the hypothesis set.
![Page 4: Multivariate linear models for regression and classification Outline: 1) multivariate linear regression 2) linear classification (perceptron) 3) logistic](https://reader036.vdocuments.net/reader036/viewer/2022082505/56649d1e5503460f949f1d2c/html5/thumbnails/4.jpg)
In logistic regression, signal become argument of a function with properties like a probability distribution
![Page 5: Multivariate linear models for regression and classification Outline: 1) multivariate linear regression 2) linear classification (perceptron) 3) logistic](https://reader036.vdocuments.net/reader036/viewer/2022082505/56649d1e5503460f949f1d2c/html5/thumbnails/5.jpg)
Objective: find w such that risk score >> 0 for patients that had a heart attack (q(s) ~ 1) and risk score << 0 for those who have not (q(s) ~ 0).
Application: risk of heart attack
![Page 6: Multivariate linear models for regression and classification Outline: 1) multivariate linear regression 2) linear classification (perceptron) 3) logistic](https://reader036.vdocuments.net/reader036/viewer/2022082505/56649d1e5503460f949f1d2c/html5/thumbnails/6.jpg)
More specifically (see text p91)
Dataset drawn from a distribution function P(y|x), which is related to hypothesis h(x) by
P(yn|xn) = h(xn) if yn = +1; P(yn|xn) = 1 - h(xn) if yn = -1
Logistic function has the property q(-s) = 1 – q(s)
Hence, both relationships are satisfied by P(yn|xn) = q(ynwT xn)
Now use maximum likelihood estimation (MLE) to derive an error function that we minimize to find the optimum w
![Page 7: Multivariate linear models for regression and classification Outline: 1) multivariate linear regression 2) linear classification (perceptron) 3) logistic](https://reader036.vdocuments.net/reader036/viewer/2022082505/56649d1e5503460f949f1d2c/html5/thumbnails/7.jpg)
Recall that MLE is used to Estimate parameters of a probability distribution given a sample X drawn from that distribution
In logistic regression, parameters are the weights
Likelihood of w given the sample Xl(w|X) = p (X |w) = ∏
t p(xt|w)
Log likelihood L(w|X) = log(l(w|X)) = ∑
t log p(xt|w)
In logistic regression, p(xt|w) = q(ynwT xn)
![Page 8: Multivariate linear models for regression and classification Outline: 1) multivariate linear regression 2) linear classification (perceptron) 3) logistic](https://reader036.vdocuments.net/reader036/viewer/2022082505/56649d1e5503460f949f1d2c/html5/thumbnails/8.jpg)
Since Log is a monotone increasing function, maximizing log(likelihood) is equivalent to minimizing -log(likelihood)
Text also normalizes by dividing by N; hence error function becomes
![Page 9: Multivariate linear models for regression and classification Outline: 1) multivariate linear regression 2) linear classification (perceptron) 3) logistic](https://reader036.vdocuments.net/reader036/viewer/2022082505/56649d1e5503460f949f1d2c/html5/thumbnails/9.jpg)
Error function of logistic regression (called cross entropy) has the desired properties.
If xn are attributes of person who has had a heart attack, wTxn >> 0 and yn > 0 so contribution to Ein(w) is small.
If xn are attributes of person who has not had a heart attack, wTxn << 0 and yn < 0 so contribution to Ein(w) is again small.
![Page 10: Multivariate linear models for regression and classification Outline: 1) multivariate linear regression 2) linear classification (perceptron) 3) logistic](https://reader036.vdocuments.net/reader036/viewer/2022082505/56649d1e5503460f949f1d2c/html5/thumbnails/10.jpg)
Error function of linear regression allows “1-step” optimization.
Not true for error function of logistic regression
Optimization is iterative; method is “steepest decent”
![Page 11: Multivariate linear models for regression and classification Outline: 1) multivariate linear regression 2) linear classification (perceptron) 3) logistic](https://reader036.vdocuments.net/reader036/viewer/2022082505/56649d1e5503460f949f1d2c/html5/thumbnails/11.jpg)
Method of steepest (gradient) decent:Fixed step size hw(1) = w(0) + hvhat
Unit vector in the direction of the gradient
![Page 12: Multivariate linear models for regression and classification Outline: 1) multivariate linear regression 2) linear classification (perceptron) 3) logistic](https://reader036.vdocuments.net/reader036/viewer/2022082505/56649d1e5503460f949f1d2c/html5/thumbnails/12.jpg)
Method of steepest (gradient) decent:Fixed leaning rate hw(1) = w(0) + delta wWeights change fastest where gradient is largest
For Ein = cross entropy, gradient is analytical
![Page 13: Multivariate linear models for regression and classification Outline: 1) multivariate linear regression 2) linear classification (perceptron) 3) logistic](https://reader036.vdocuments.net/reader036/viewer/2022082505/56649d1e5503460f949f1d2c/html5/thumbnails/13.jpg)
Logistics regression algorithm
![Page 14: Multivariate linear models for regression and classification Outline: 1) multivariate linear regression 2) linear classification (perceptron) 3) logistic](https://reader036.vdocuments.net/reader036/viewer/2022082505/56649d1e5503460f949f1d2c/html5/thumbnails/14.jpg)
How to compute gradient of Ein
![Page 15: Multivariate linear models for regression and classification Outline: 1) multivariate linear regression 2) linear classification (perceptron) 3) logistic](https://reader036.vdocuments.net/reader036/viewer/2022082505/56649d1e5503460f949f1d2c/html5/thumbnails/15.jpg)
How to known when to stop
![Page 16: Multivariate linear models for regression and classification Outline: 1) multivariate linear regression 2) linear classification (perceptron) 3) logistic](https://reader036.vdocuments.net/reader036/viewer/2022082505/56649d1e5503460f949f1d2c/html5/thumbnails/16.jpg)
Assignment 6: Due 10-30-14