chris&bishop,&prml pattern&recognition& and machine&learning€¦ ·...
TRANSCRIPT
![Page 1: Chris&Bishop,&PRML PATTERN&RECOGNITION& AND MACHINE&LEARNING€¦ · Chris&Bishop,&PRML PATTERN&RECOGNITION& ANDMACHINE&LEARNING CH&1&2:&INTROTO&PROBABILITY&DENSITY&ESTIMATION COMPSCI'591/691NR](https://reader033.vdocuments.net/reader033/viewer/2022042712/5f98cc31adc0b8014602511c/html5/thumbnails/1.jpg)
Chris&Bishop,&PRML
PATTERN&RECOGNITION&AND MACHINE&LEARNINGCH&1&2:&INTRO TO&PROBABILITY&DENSITY&ESTIMATION
COMPSCI'591/691NR'Neural'Networks'and'Neurodynamics
1/22/2020
![Page 2: Chris&Bishop,&PRML PATTERN&RECOGNITION& AND MACHINE&LEARNING€¦ · Chris&Bishop,&PRML PATTERN&RECOGNITION& ANDMACHINE&LEARNING CH&1&2:&INTROTO&PROBABILITY&DENSITY&ESTIMATION COMPSCI'591/691NR](https://reader033.vdocuments.net/reader033/viewer/2022042712/5f98cc31adc0b8014602511c/html5/thumbnails/2.jpg)
Example
Handwritten/Digit/Recognition
![Page 3: Chris&Bishop,&PRML PATTERN&RECOGNITION& AND MACHINE&LEARNING€¦ · Chris&Bishop,&PRML PATTERN&RECOGNITION& ANDMACHINE&LEARNING CH&1&2:&INTROTO&PROBABILITY&DENSITY&ESTIMATION COMPSCI'591/691NR](https://reader033.vdocuments.net/reader033/viewer/2022042712/5f98cc31adc0b8014602511c/html5/thumbnails/3.jpg)
Character(Recognition
Handwritten(characters:(a(and(bExamples(in(the(data(set((e.g.(50(of(each)Classification(task((2(classes)
Humans:(easy(by(observingComputers:
Require(encoding(into(numerical(stringsDevelop(a(classification(algorithm((parametric(or(nonFparametric)Convert(highFdimensional(data(space(into(lowFdim(feature(space((see(next.)
![Page 4: Chris&Bishop,&PRML PATTERN&RECOGNITION& AND MACHINE&LEARNING€¦ · Chris&Bishop,&PRML PATTERN&RECOGNITION& ANDMACHINE&LEARNING CH&1&2:&INTROTO&PROBABILITY&DENSITY&ESTIMATION COMPSCI'591/691NR](https://reader033.vdocuments.net/reader033/viewer/2022042712/5f98cc31adc0b8014602511c/html5/thumbnails/4.jpg)
Parametric)Classification)by)machines)3example
Calculate)the)aspect)ratioR)=)height/width
Modeling)assumption: R(b))>)R(a)It’s)true)in)most)of)the)cases)(not)always!)
Perform)classification)based)on)the)modelNeeds)a)decision)criterion)Calculate)error)rate)of)classificationC)=)#)misclassified)/#)total)samples
![Page 5: Chris&Bishop,&PRML PATTERN&RECOGNITION& AND MACHINE&LEARNING€¦ · Chris&Bishop,&PRML PATTERN&RECOGNITION& ANDMACHINE&LEARNING CH&1&2:&INTROTO&PROBABILITY&DENSITY&ESTIMATION COMPSCI'591/691NR](https://reader033.vdocuments.net/reader033/viewer/2022042712/5f98cc31adc0b8014602511c/html5/thumbnails/5.jpg)
Classification+(cont’d)
Develop+a+good+classifier+With+low+error+rate+CE.g.,+calculate+class+centers+and+use+decision+surface+at+the+middle+line+between+them
We+will+see+this+approach+is+optimal+in+some+senseExpecting+good+results+for+new+examples
@@>+good+GENERALIZATION+!!Not+easy
![Page 6: Chris&Bishop,&PRML PATTERN&RECOGNITION& AND MACHINE&LEARNING€¦ · Chris&Bishop,&PRML PATTERN&RECOGNITION& ANDMACHINE&LEARNING CH&1&2:&INTROTO&PROBABILITY&DENSITY&ESTIMATION COMPSCI'591/691NR](https://reader033.vdocuments.net/reader033/viewer/2022042712/5f98cc31adc0b8014602511c/html5/thumbnails/6.jpg)
Higher'dimensional'(dim>1)'feature'spaces
Decision'along'not'just'a'threshold'but'a'surface':'a'subspace'of'co<dimension'1'in'the'feature'space
Can'be'linear'discriminantIn'this'case'the'decision'surface'is'a'(high<dimensional)'plane
More'complicated:'nonlinear'discriminant(statistical'model,'or'model<free'NN)
![Page 7: Chris&Bishop,&PRML PATTERN&RECOGNITION& AND MACHINE&LEARNING€¦ · Chris&Bishop,&PRML PATTERN&RECOGNITION& ANDMACHINE&LEARNING CH&1&2:&INTROTO&PROBABILITY&DENSITY&ESTIMATION COMPSCI'591/691NR](https://reader033.vdocuments.net/reader033/viewer/2022042712/5f98cc31adc0b8014602511c/html5/thumbnails/7.jpg)
Classification+vs+prediction
The+two+main+problems+in+NN+studiesNow+we+concentrate+on+classificationLater+discuss+regression+and+prediction
Not+independent+problem
![Page 8: Chris&Bishop,&PRML PATTERN&RECOGNITION& AND MACHINE&LEARNING€¦ · Chris&Bishop,&PRML PATTERN&RECOGNITION& ANDMACHINE&LEARNING CH&1&2:&INTROTO&PROBABILITY&DENSITY&ESTIMATION COMPSCI'591/691NR](https://reader033.vdocuments.net/reader033/viewer/2022042712/5f98cc31adc0b8014602511c/html5/thumbnails/8.jpg)
Preprocessing
Feature/extractionEg/normalizationData/compression/(decrease/dimensionality)Get/rid/off/drift/and/other/unwanted/effectsBe/aware!!/Don’t/destroy/the/data
You/may/think/something/is/not/important/but/in/fact/it/is!!
Preprocessing/often/a/matter/of/‘art’/and/it/is/difficult/to/give/general/rules
Must/think!!/Proper/preprocessing/is/often/a/key/to/successful/classification...
![Page 9: Chris&Bishop,&PRML PATTERN&RECOGNITION& AND MACHINE&LEARNING€¦ · Chris&Bishop,&PRML PATTERN&RECOGNITION& ANDMACHINE&LEARNING CH&1&2:&INTROTO&PROBABILITY&DENSITY&ESTIMATION COMPSCI'591/691NR](https://reader033.vdocuments.net/reader033/viewer/2022042712/5f98cc31adc0b8014602511c/html5/thumbnails/9.jpg)
Statistical(Pattern(Recognition(versus(NNs
Statistical(classifier((feature(extraction)
Neural(network(classifier
Input(high dim)Preprocessing(normalize,…)
0111010101000010101000100001111100101
1111010101000010101000100110001111000
01010000011110001010
0 / 1
Compression
Output (class)Yes/No / a or b etc.
NNStat. Discr.Bayes
![Page 10: Chris&Bishop,&PRML PATTERN&RECOGNITION& AND MACHINE&LEARNING€¦ · Chris&Bishop,&PRML PATTERN&RECOGNITION& ANDMACHINE&LEARNING CH&1&2:&INTROTO&PROBABILITY&DENSITY&ESTIMATION COMPSCI'591/691NR](https://reader033.vdocuments.net/reader033/viewer/2022042712/5f98cc31adc0b8014602511c/html5/thumbnails/10.jpg)
Example:)Curve)fitting
Given:)x)and)t)vectorsEg,)x=1:100t)function)is)known)at)given)(time))instances
Goal:)fit)this)curve)and)get)y)at)any)value)(not)only)the)given)100)points)
Expect)good)results)in)apices)x(i):)y(x(i)))≈)t(i)Error)function:)sum)of)squared)errors!!Also)desired)good)approximation)at)points)in)between)interpolation and)also)extrapolation
![Page 11: Chris&Bishop,&PRML PATTERN&RECOGNITION& AND MACHINE&LEARNING€¦ · Chris&Bishop,&PRML PATTERN&RECOGNITION& ANDMACHINE&LEARNING CH&1&2:&INTROTO&PROBABILITY&DENSITY&ESTIMATION COMPSCI'591/691NR](https://reader033.vdocuments.net/reader033/viewer/2022042712/5f98cc31adc0b8014602511c/html5/thumbnails/11.jpg)
Polynomial)Curve)Fitting
![Page 12: Chris&Bishop,&PRML PATTERN&RECOGNITION& AND MACHINE&LEARNING€¦ · Chris&Bishop,&PRML PATTERN&RECOGNITION& ANDMACHINE&LEARNING CH&1&2:&INTROTO&PROBABILITY&DENSITY&ESTIMATION COMPSCI'591/691NR](https://reader033.vdocuments.net/reader033/viewer/2022042712/5f98cc31adc0b8014602511c/html5/thumbnails/12.jpg)
Error$Function
Sum$of$squared$errorsIt$is$a$metric$for$the$goodness$of$the$modelDistance$in$the$Euclidean$space$(N$dim)
We$want$to$adjust$the$parameters$of$the$model$(eg,$coeff’s$of$the$polynomial)$that$minimize$this$error
NB:$there$are$more$complicated$error$functions$but$SSE$is$optimal$in$some$sense$(see$later)
![Page 13: Chris&Bishop,&PRML PATTERN&RECOGNITION& AND MACHINE&LEARNING€¦ · Chris&Bishop,&PRML PATTERN&RECOGNITION& ANDMACHINE&LEARNING CH&1&2:&INTROTO&PROBABILITY&DENSITY&ESTIMATION COMPSCI'591/691NR](https://reader033.vdocuments.net/reader033/viewer/2022042712/5f98cc31adc0b8014602511c/html5/thumbnails/13.jpg)
Sum$of$Squares,Error,Function
![Page 14: Chris&Bishop,&PRML PATTERN&RECOGNITION& AND MACHINE&LEARNING€¦ · Chris&Bishop,&PRML PATTERN&RECOGNITION& ANDMACHINE&LEARNING CH&1&2:&INTROTO&PROBABILITY&DENSITY&ESTIMATION COMPSCI'591/691NR](https://reader033.vdocuments.net/reader033/viewer/2022042712/5f98cc31adc0b8014602511c/html5/thumbnails/14.jpg)
Generalization
Optimum/model/complexity!!!By/following/too/closely/the/training/data/
you/won’t/be/able/to/perform/well/in/new/data/==>/meaning/bad/generalization
You/don’t/want/to/learn/the/noise/noise/is/inherently/present/in/life/Eg,/handwriting:/slipping/of/pen,/tired,/etc...
![Page 15: Chris&Bishop,&PRML PATTERN&RECOGNITION& AND MACHINE&LEARNING€¦ · Chris&Bishop,&PRML PATTERN&RECOGNITION& ANDMACHINE&LEARNING CH&1&2:&INTROTO&PROBABILITY&DENSITY&ESTIMATION COMPSCI'591/691NR](https://reader033.vdocuments.net/reader033/viewer/2022042712/5f98cc31adc0b8014602511c/html5/thumbnails/15.jpg)
0th Order(Polynomial
![Page 16: Chris&Bishop,&PRML PATTERN&RECOGNITION& AND MACHINE&LEARNING€¦ · Chris&Bishop,&PRML PATTERN&RECOGNITION& ANDMACHINE&LEARNING CH&1&2:&INTROTO&PROBABILITY&DENSITY&ESTIMATION COMPSCI'591/691NR](https://reader033.vdocuments.net/reader033/viewer/2022042712/5f98cc31adc0b8014602511c/html5/thumbnails/16.jpg)
1st Order(Polynomial
![Page 17: Chris&Bishop,&PRML PATTERN&RECOGNITION& AND MACHINE&LEARNING€¦ · Chris&Bishop,&PRML PATTERN&RECOGNITION& ANDMACHINE&LEARNING CH&1&2:&INTROTO&PROBABILITY&DENSITY&ESTIMATION COMPSCI'591/691NR](https://reader033.vdocuments.net/reader033/viewer/2022042712/5f98cc31adc0b8014602511c/html5/thumbnails/17.jpg)
3rd Order&Polynomial
![Page 18: Chris&Bishop,&PRML PATTERN&RECOGNITION& AND MACHINE&LEARNING€¦ · Chris&Bishop,&PRML PATTERN&RECOGNITION& ANDMACHINE&LEARNING CH&1&2:&INTROTO&PROBABILITY&DENSITY&ESTIMATION COMPSCI'591/691NR](https://reader033.vdocuments.net/reader033/viewer/2022042712/5f98cc31adc0b8014602511c/html5/thumbnails/18.jpg)
9th Order(Polynomial
![Page 19: Chris&Bishop,&PRML PATTERN&RECOGNITION& AND MACHINE&LEARNING€¦ · Chris&Bishop,&PRML PATTERN&RECOGNITION& ANDMACHINE&LEARNING CH&1&2:&INTROTO&PROBABILITY&DENSITY&ESTIMATION COMPSCI'591/691NR](https://reader033.vdocuments.net/reader033/viewer/2022042712/5f98cc31adc0b8014602511c/html5/thumbnails/19.jpg)
Over%fitting
Root%Mean%Square2(RMS)2Error:
![Page 20: Chris&Bishop,&PRML PATTERN&RECOGNITION& AND MACHINE&LEARNING€¦ · Chris&Bishop,&PRML PATTERN&RECOGNITION& ANDMACHINE&LEARNING CH&1&2:&INTROTO&PROBABILITY&DENSITY&ESTIMATION COMPSCI'591/691NR](https://reader033.vdocuments.net/reader033/viewer/2022042712/5f98cc31adc0b8014602511c/html5/thumbnails/20.jpg)
Polynomial)Coefficients)))
![Page 21: Chris&Bishop,&PRML PATTERN&RECOGNITION& AND MACHINE&LEARNING€¦ · Chris&Bishop,&PRML PATTERN&RECOGNITION& ANDMACHINE&LEARNING CH&1&2:&INTROTO&PROBABILITY&DENSITY&ESTIMATION COMPSCI'591/691NR](https://reader033.vdocuments.net/reader033/viewer/2022042712/5f98cc31adc0b8014602511c/html5/thumbnails/21.jpg)
Data$Set$Size:$9th Order$Polynomial
![Page 22: Chris&Bishop,&PRML PATTERN&RECOGNITION& AND MACHINE&LEARNING€¦ · Chris&Bishop,&PRML PATTERN&RECOGNITION& ANDMACHINE&LEARNING CH&1&2:&INTROTO&PROBABILITY&DENSITY&ESTIMATION COMPSCI'591/691NR](https://reader033.vdocuments.net/reader033/viewer/2022042712/5f98cc31adc0b8014602511c/html5/thumbnails/22.jpg)
Data$Set$Size:$9th Order$Polynomial
![Page 23: Chris&Bishop,&PRML PATTERN&RECOGNITION& AND MACHINE&LEARNING€¦ · Chris&Bishop,&PRML PATTERN&RECOGNITION& ANDMACHINE&LEARNING CH&1&2:&INTROTO&PROBABILITY&DENSITY&ESTIMATION COMPSCI'591/691NR](https://reader033.vdocuments.net/reader033/viewer/2022042712/5f98cc31adc0b8014602511c/html5/thumbnails/23.jpg)
Regularization
Penalize.large.coefficient.values
![Page 24: Chris&Bishop,&PRML PATTERN&RECOGNITION& AND MACHINE&LEARNING€¦ · Chris&Bishop,&PRML PATTERN&RECOGNITION& ANDMACHINE&LEARNING CH&1&2:&INTROTO&PROBABILITY&DENSITY&ESTIMATION COMPSCI'591/691NR](https://reader033.vdocuments.net/reader033/viewer/2022042712/5f98cc31adc0b8014602511c/html5/thumbnails/24.jpg)
Regularization:.
![Page 25: Chris&Bishop,&PRML PATTERN&RECOGNITION& AND MACHINE&LEARNING€¦ · Chris&Bishop,&PRML PATTERN&RECOGNITION& ANDMACHINE&LEARNING CH&1&2:&INTROTO&PROBABILITY&DENSITY&ESTIMATION COMPSCI'591/691NR](https://reader033.vdocuments.net/reader033/viewer/2022042712/5f98cc31adc0b8014602511c/html5/thumbnails/25.jpg)
Regularization:.
![Page 26: Chris&Bishop,&PRML PATTERN&RECOGNITION& AND MACHINE&LEARNING€¦ · Chris&Bishop,&PRML PATTERN&RECOGNITION& ANDMACHINE&LEARNING CH&1&2:&INTROTO&PROBABILITY&DENSITY&ESTIMATION COMPSCI'591/691NR](https://reader033.vdocuments.net/reader033/viewer/2022042712/5f98cc31adc0b8014602511c/html5/thumbnails/26.jpg)
Regularization:...........vs..
![Page 27: Chris&Bishop,&PRML PATTERN&RECOGNITION& AND MACHINE&LEARNING€¦ · Chris&Bishop,&PRML PATTERN&RECOGNITION& ANDMACHINE&LEARNING CH&1&2:&INTROTO&PROBABILITY&DENSITY&ESTIMATION COMPSCI'591/691NR](https://reader033.vdocuments.net/reader033/viewer/2022042712/5f98cc31adc0b8014602511c/html5/thumbnails/27.jpg)
Polynomial)Coefficients)))
![Page 28: Chris&Bishop,&PRML PATTERN&RECOGNITION& AND MACHINE&LEARNING€¦ · Chris&Bishop,&PRML PATTERN&RECOGNITION& ANDMACHINE&LEARNING CH&1&2:&INTROTO&PROBABILITY&DENSITY&ESTIMATION COMPSCI'591/691NR](https://reader033.vdocuments.net/reader033/viewer/2022042712/5f98cc31adc0b8014602511c/html5/thumbnails/28.jpg)
The$Rules$of$Probability
Sum$Rule
Product$Rule
![Page 29: Chris&Bishop,&PRML PATTERN&RECOGNITION& AND MACHINE&LEARNING€¦ · Chris&Bishop,&PRML PATTERN&RECOGNITION& ANDMACHINE&LEARNING CH&1&2:&INTROTO&PROBABILITY&DENSITY&ESTIMATION COMPSCI'591/691NR](https://reader033.vdocuments.net/reader033/viewer/2022042712/5f98cc31adc0b8014602511c/html5/thumbnails/29.jpg)
Bayes&Theorem
Probability:P(A)&5 the&probability&of&occurrence&of&A
Head/tail:&1/2,&dice:&1/6,&etc.
Conditional&probability:P(A|B)&5 probability&of&A,&assuming&B
P(3|uneven)=1/3
Bayes&formula:P(A|B)P(B)&=&P(B|A)P(A)&
Based&on&joint&probabilities
![Page 30: Chris&Bishop,&PRML PATTERN&RECOGNITION& AND MACHINE&LEARNING€¦ · Chris&Bishop,&PRML PATTERN&RECOGNITION& ANDMACHINE&LEARNING CH&1&2:&INTROTO&PROBABILITY&DENSITY&ESTIMATION COMPSCI'591/691NR](https://reader033.vdocuments.net/reader033/viewer/2022042712/5f98cc31adc0b8014602511c/html5/thumbnails/30.jpg)
Interpretation*of*Bayes*Theorem
X*3 sample*exampleC_i*3 denotes*the*i3th*class*(i=1,*2)
P(C_i|X))=)P(X|C_i)P(C_i))/)P(X)P(C_i|X)*3 posterior*probability*of*X*in*class*iP(X|C_i)*3 class3conditional*probabilityP(C_i)****3 prior*probability*of*class*C_iP(X)*******3 probability*of*observing*X*in*general
Classification*task:*determine*P(C_i|X)*!!
![Page 31: Chris&Bishop,&PRML PATTERN&RECOGNITION& AND MACHINE&LEARNING€¦ · Chris&Bishop,&PRML PATTERN&RECOGNITION& ANDMACHINE&LEARNING CH&1&2:&INTROTO&PROBABILITY&DENSITY&ESTIMATION COMPSCI'591/691NR](https://reader033.vdocuments.net/reader033/viewer/2022042712/5f98cc31adc0b8014602511c/html5/thumbnails/31.jpg)
Classification:,using,Bayes,
Determine,or,estimate,RHS,probabilitiesClass:conditional,,prior,,probability,of,the,event
Use,Bayes,to,evaluate,posterior,probabilityDecision,criteria/,discriminant,function:
Assign,class,k,if,P(C_k|X))>)P(C_j|X) valid,,,for,all,j,≠,k,
Maximum,likelihood,discriminant,function
![Page 32: Chris&Bishop,&PRML PATTERN&RECOGNITION& AND MACHINE&LEARNING€¦ · Chris&Bishop,&PRML PATTERN&RECOGNITION& ANDMACHINE&LEARNING CH&1&2:&INTROTO&PROBABILITY&DENSITY&ESTIMATION COMPSCI'591/691NR](https://reader033.vdocuments.net/reader033/viewer/2022042712/5f98cc31adc0b8014602511c/html5/thumbnails/32.jpg)
Bayes’s'decision'criterion/ruleQ:'what'is'the'best'decision'surface'in'a'general'nonlinear'classification'problem?
Two'steps:1.'Inference:'determine'the'‘posterior'probabilities’'P(C_k|X);'k=1,2'(2'classes)
2.'Decision:'assign'the'actual'input'X'to'one'of'the'classesUnder'certain'plausible'statistical'conditions'choosing'the'max'posterior'prob'is'the'best'as'it'minimizes'the'classification'error'SSE
![Page 33: Chris&Bishop,&PRML PATTERN&RECOGNITION& AND MACHINE&LEARNING€¦ · Chris&Bishop,&PRML PATTERN&RECOGNITION& ANDMACHINE&LEARNING CH&1&2:&INTROTO&PROBABILITY&DENSITY&ESTIMATION COMPSCI'591/691NR](https://reader033.vdocuments.net/reader033/viewer/2022042712/5f98cc31adc0b8014602511c/html5/thumbnails/33.jpg)
Implementing*the*Bayes*optimal*decision*rule*in*NNs
Assume*NN*is*a*black*boxInput*XOutput:*
approximates*posterior*probability!!Classification:
Choose*the*output*class*node*with*the*max*value*and*that*is*the*best*choice!
This*is*the*beauty*of*NNs!*BUT:*be*aware:*there*is*no*gain*without*pain...
![Page 34: Chris&Bishop,&PRML PATTERN&RECOGNITION& AND MACHINE&LEARNING€¦ · Chris&Bishop,&PRML PATTERN&RECOGNITION& ANDMACHINE&LEARNING CH&1&2:&INTROTO&PROBABILITY&DENSITY&ESTIMATION COMPSCI'591/691NR](https://reader033.vdocuments.net/reader033/viewer/2022042712/5f98cc31adc0b8014602511c/html5/thumbnails/34.jpg)
Probability*Density*Estimation*Methods
1. Parametric*methods❂ Normal*distribution❂ Determine*parameters:*using*maximum*likelihood❂ Inference*in*Bayesian*approach
2. NonBparametric*methods❂ Histograms❂ Kernel*methods❂ K*nearest*neighbors
3. SemiBparametric❂ Mixture*models*(also*NNs)
![Page 35: Chris&Bishop,&PRML PATTERN&RECOGNITION& AND MACHINE&LEARNING€¦ · Chris&Bishop,&PRML PATTERN&RECOGNITION& ANDMACHINE&LEARNING CH&1&2:&INTROTO&PROBABILITY&DENSITY&ESTIMATION COMPSCI'591/691NR](https://reader033.vdocuments.net/reader033/viewer/2022042712/5f98cc31adc0b8014602511c/html5/thumbnails/35.jpg)
Advantages*of*Normal*Distribution
1. Simple*analytical*form2. It*is*common*in*practice*due*to*CLT3. For*any*nonsingular*linear*transformation*the*distance*
remains*positive*definite*and*of*quadratic*form*=>*remains*normal*distribution
4. Marginal*densities*(integrated*over*some*variables]*are*normal
5. Conditional*densities*(fixing*some*variables]*are*normal6. Exists*a*LINEAR*transformation*that*diagonalizes*
covariance*matrix*,*and*in*this*new*system*the*variables*are*independent*(factorized*to*components].
7. Normal*density*maximizes*entropy*for*some*µ and*!.
![Page 36: Chris&Bishop,&PRML PATTERN&RECOGNITION& AND MACHINE&LEARNING€¦ · Chris&Bishop,&PRML PATTERN&RECOGNITION& ANDMACHINE&LEARNING CH&1&2:&INTROTO&PROBABILITY&DENSITY&ESTIMATION COMPSCI'591/691NR](https://reader033.vdocuments.net/reader033/viewer/2022042712/5f98cc31adc0b8014602511c/html5/thumbnails/36.jpg)
The$Gaussian$Distribution
![Page 37: Chris&Bishop,&PRML PATTERN&RECOGNITION& AND MACHINE&LEARNING€¦ · Chris&Bishop,&PRML PATTERN&RECOGNITION& ANDMACHINE&LEARNING CH&1&2:&INTROTO&PROBABILITY&DENSITY&ESTIMATION COMPSCI'591/691NR](https://reader033.vdocuments.net/reader033/viewer/2022042712/5f98cc31adc0b8014602511c/html5/thumbnails/37.jpg)
Gaussian'Mean'and'Variance
![Page 38: Chris&Bishop,&PRML PATTERN&RECOGNITION& AND MACHINE&LEARNING€¦ · Chris&Bishop,&PRML PATTERN&RECOGNITION& ANDMACHINE&LEARNING CH&1&2:&INTROTO&PROBABILITY&DENSITY&ESTIMATION COMPSCI'591/691NR](https://reader033.vdocuments.net/reader033/viewer/2022042712/5f98cc31adc0b8014602511c/html5/thumbnails/38.jpg)
The$Multivariate$Gaussian
![Page 39: Chris&Bishop,&PRML PATTERN&RECOGNITION& AND MACHINE&LEARNING€¦ · Chris&Bishop,&PRML PATTERN&RECOGNITION& ANDMACHINE&LEARNING CH&1&2:&INTROTO&PROBABILITY&DENSITY&ESTIMATION COMPSCI'591/691NR](https://reader033.vdocuments.net/reader033/viewer/2022042712/5f98cc31adc0b8014602511c/html5/thumbnails/39.jpg)
Gaussian'Parameter'Estimation
Likelihood'function
![Page 40: Chris&Bishop,&PRML PATTERN&RECOGNITION& AND MACHINE&LEARNING€¦ · Chris&Bishop,&PRML PATTERN&RECOGNITION& ANDMACHINE&LEARNING CH&1&2:&INTROTO&PROBABILITY&DENSITY&ESTIMATION COMPSCI'591/691NR](https://reader033.vdocuments.net/reader033/viewer/2022042712/5f98cc31adc0b8014602511c/html5/thumbnails/40.jpg)
Discriminant*Function*for*GaussianOur*discriminant*function*is
yk =*log*p(x|C_k)*+*log*P(C_k)substitute*the*normal*functionyk =*(xCµ!)T"!C1*(xCµ!)*–1/2*log[|"k|]*+*log*P(C_k)
Assume:Covariances*are*independent*of*k*classesThen*2nd term*is*classCindependent*and*also*quadratic*term*in*x*is*classCindependent
Covariance*and*inverse*is*symmetric:yk =*WT
k*x*+*wk0 CC this*is*simple*linear*form!!!*…derive!Voronoi*tessallation*Here*WT
k*=*µ!T"C1**and*wk0*=*C1/2µ!T"C1*µ! +*log*P(C_k)
![Page 41: Chris&Bishop,&PRML PATTERN&RECOGNITION& AND MACHINE&LEARNING€¦ · Chris&Bishop,&PRML PATTERN&RECOGNITION& ANDMACHINE&LEARNING CH&1&2:&INTROTO&PROBABILITY&DENSITY&ESTIMATION COMPSCI'591/691NR](https://reader033.vdocuments.net/reader033/viewer/2022042712/5f98cc31adc0b8014602511c/html5/thumbnails/41.jpg)
Prototypes/*Template*Matching
If*the*following*conditions*are*satisfied:k;class*problem*with*identical*! covariancesIndependent*variables*(I.e,*diagonal*!]We*can*drop*the*constant*term*of*discriminantyk =*;||x;µ"||2*/2#2 +*log*P(C_k)
Discriminant*functionNearest*class*center*will*be*chosen!As*measured*by*Euclidean*distance
![Page 42: Chris&Bishop,&PRML PATTERN&RECOGNITION& AND MACHINE&LEARNING€¦ · Chris&Bishop,&PRML PATTERN&RECOGNITION& ANDMACHINE&LEARNING CH&1&2:&INTROTO&PROBABILITY&DENSITY&ESTIMATION COMPSCI'591/691NR](https://reader033.vdocuments.net/reader033/viewer/2022042712/5f98cc31adc0b8014602511c/html5/thumbnails/42.jpg)
Model&optimization:Maximum&Likelihood&vs&Bayesian&Inference
Maximum&likelihood:Maximize&the&value&of&the&likelihood&function&based&on&training&data
E.g.,&yk&as&derived&earlierBayesian&Inference
Estimate&the¶meters&of&the&probability&density&function&
Initial&set&prior&distributionUse&Bayes&Theorem&to&convert&to&posterior&distributionFinal&probability&density:&integrate&over&all&possible&values&of¶meters&weighted&by&the&posterior&probability
![Page 43: Chris&Bishop,&PRML PATTERN&RECOGNITION& AND MACHINE&LEARNING€¦ · Chris&Bishop,&PRML PATTERN&RECOGNITION& ANDMACHINE&LEARNING CH&1&2:&INTROTO&PROBABILITY&DENSITY&ESTIMATION COMPSCI'591/691NR](https://reader033.vdocuments.net/reader033/viewer/2022042712/5f98cc31adc0b8014602511c/html5/thumbnails/43.jpg)
Nonparametric,Methods
HistogramCalculate,by,dividing,into,n9binsThis,is,a,simple,estimation,of,probability,distribution
Can,be,normal,distributionOr,multi9modal,normal,,etc.
Assuming,p(x),is,approx,constantp(x),=,K/NVN,data,points,,K,is,within,a,region,R,,V,is,volume,of,R
![Page 44: Chris&Bishop,&PRML PATTERN&RECOGNITION& AND MACHINE&LEARNING€¦ · Chris&Bishop,&PRML PATTERN&RECOGNITION& ANDMACHINE&LEARNING CH&1&2:&INTROTO&PROBABILITY&DENSITY&ESTIMATION COMPSCI'591/691NR](https://reader033.vdocuments.net/reader033/viewer/2022042712/5f98cc31adc0b8014602511c/html5/thumbnails/44.jpg)
Curve&Fitting&Re-visited
![Page 45: Chris&Bishop,&PRML PATTERN&RECOGNITION& AND MACHINE&LEARNING€¦ · Chris&Bishop,&PRML PATTERN&RECOGNITION& ANDMACHINE&LEARNING CH&1&2:&INTROTO&PROBABILITY&DENSITY&ESTIMATION COMPSCI'591/691NR](https://reader033.vdocuments.net/reader033/viewer/2022042712/5f98cc31adc0b8014602511c/html5/thumbnails/45.jpg)
Maximum'Likelihood
Determine''''''''''''by'minimizing'sum8of8squares'error,'''''''''''''.
![Page 46: Chris&Bishop,&PRML PATTERN&RECOGNITION& AND MACHINE&LEARNING€¦ · Chris&Bishop,&PRML PATTERN&RECOGNITION& ANDMACHINE&LEARNING CH&1&2:&INTROTO&PROBABILITY&DENSITY&ESTIMATION COMPSCI'591/691NR](https://reader033.vdocuments.net/reader033/viewer/2022042712/5f98cc31adc0b8014602511c/html5/thumbnails/46.jpg)
Model&Selection
Cross/Validation
![Page 47: Chris&Bishop,&PRML PATTERN&RECOGNITION& AND MACHINE&LEARNING€¦ · Chris&Bishop,&PRML PATTERN&RECOGNITION& ANDMACHINE&LEARNING CH&1&2:&INTROTO&PROBABILITY&DENSITY&ESTIMATION COMPSCI'591/691NR](https://reader033.vdocuments.net/reader033/viewer/2022042712/5f98cc31adc0b8014602511c/html5/thumbnails/47.jpg)
Curse&of&Dimensionality
![Page 48: Chris&Bishop,&PRML PATTERN&RECOGNITION& AND MACHINE&LEARNING€¦ · Chris&Bishop,&PRML PATTERN&RECOGNITION& ANDMACHINE&LEARNING CH&1&2:&INTROTO&PROBABILITY&DENSITY&ESTIMATION COMPSCI'591/691NR](https://reader033.vdocuments.net/reader033/viewer/2022042712/5f98cc31adc0b8014602511c/html5/thumbnails/48.jpg)
Decision(Theory
Inference(stepDetermine(either((((((((((((or(((((((((((.
Decision(stepFor(given(x,(determine(optimal(t.
![Page 49: Chris&Bishop,&PRML PATTERN&RECOGNITION& AND MACHINE&LEARNING€¦ · Chris&Bishop,&PRML PATTERN&RECOGNITION& ANDMACHINE&LEARNING CH&1&2:&INTROTO&PROBABILITY&DENSITY&ESTIMATION COMPSCI'591/691NR](https://reader033.vdocuments.net/reader033/viewer/2022042712/5f98cc31adc0b8014602511c/html5/thumbnails/49.jpg)
Minimum&Misclassification&Rate
![Page 50: Chris&Bishop,&PRML PATTERN&RECOGNITION& AND MACHINE&LEARNING€¦ · Chris&Bishop,&PRML PATTERN&RECOGNITION& ANDMACHINE&LEARNING CH&1&2:&INTROTO&PROBABILITY&DENSITY&ESTIMATION COMPSCI'591/691NR](https://reader033.vdocuments.net/reader033/viewer/2022042712/5f98cc31adc0b8014602511c/html5/thumbnails/50.jpg)
Minimum&Expected&Loss
Regions&&&&&&&are&chosen&to&minimize
![Page 51: Chris&Bishop,&PRML PATTERN&RECOGNITION& AND MACHINE&LEARNING€¦ · Chris&Bishop,&PRML PATTERN&RECOGNITION& ANDMACHINE&LEARNING CH&1&2:&INTROTO&PROBABILITY&DENSITY&ESTIMATION COMPSCI'591/691NR](https://reader033.vdocuments.net/reader033/viewer/2022042712/5f98cc31adc0b8014602511c/html5/thumbnails/51.jpg)
Generative)vs)Discriminative
Generative)approach:)ModelUse)Bayes’)theorem
Discriminative)approach:)Model)))))))))))directly
![Page 52: Chris&Bishop,&PRML PATTERN&RECOGNITION& AND MACHINE&LEARNING€¦ · Chris&Bishop,&PRML PATTERN&RECOGNITION& ANDMACHINE&LEARNING CH&1&2:&INTROTO&PROBABILITY&DENSITY&ESTIMATION COMPSCI'591/691NR](https://reader033.vdocuments.net/reader033/viewer/2022042712/5f98cc31adc0b8014602511c/html5/thumbnails/52.jpg)
Entropy
Important+quantity+in• coding+theory• statistical+physics•machine+learning
![Page 53: Chris&Bishop,&PRML PATTERN&RECOGNITION& AND MACHINE&LEARNING€¦ · Chris&Bishop,&PRML PATTERN&RECOGNITION& ANDMACHINE&LEARNING CH&1&2:&INTROTO&PROBABILITY&DENSITY&ESTIMATION COMPSCI'591/691NR](https://reader033.vdocuments.net/reader033/viewer/2022042712/5f98cc31adc0b8014602511c/html5/thumbnails/53.jpg)
Entropy
Coding,theory:,x discrete,with,8,possible,states;,how,many,bits,to,transmit,the,state,of,x?
All,states,equally,likely
![Page 54: Chris&Bishop,&PRML PATTERN&RECOGNITION& AND MACHINE&LEARNING€¦ · Chris&Bishop,&PRML PATTERN&RECOGNITION& ANDMACHINE&LEARNING CH&1&2:&INTROTO&PROBABILITY&DENSITY&ESTIMATION COMPSCI'591/691NR](https://reader033.vdocuments.net/reader033/viewer/2022042712/5f98cc31adc0b8014602511c/html5/thumbnails/54.jpg)
Entropy
![Page 55: Chris&Bishop,&PRML PATTERN&RECOGNITION& AND MACHINE&LEARNING€¦ · Chris&Bishop,&PRML PATTERN&RECOGNITION& ANDMACHINE&LEARNING CH&1&2:&INTROTO&PROBABILITY&DENSITY&ESTIMATION COMPSCI'591/691NR](https://reader033.vdocuments.net/reader033/viewer/2022042712/5f98cc31adc0b8014602511c/html5/thumbnails/55.jpg)
Entropy
In)how)many)ways)can)N identical)objects)be)allocated)Mbins?
Entropy)maximized)when
![Page 56: Chris&Bishop,&PRML PATTERN&RECOGNITION& AND MACHINE&LEARNING€¦ · Chris&Bishop,&PRML PATTERN&RECOGNITION& ANDMACHINE&LEARNING CH&1&2:&INTROTO&PROBABILITY&DENSITY&ESTIMATION COMPSCI'591/691NR](https://reader033.vdocuments.net/reader033/viewer/2022042712/5f98cc31adc0b8014602511c/html5/thumbnails/56.jpg)
Entropy
![Page 57: Chris&Bishop,&PRML PATTERN&RECOGNITION& AND MACHINE&LEARNING€¦ · Chris&Bishop,&PRML PATTERN&RECOGNITION& ANDMACHINE&LEARNING CH&1&2:&INTROTO&PROBABILITY&DENSITY&ESTIMATION COMPSCI'591/691NR](https://reader033.vdocuments.net/reader033/viewer/2022042712/5f98cc31adc0b8014602511c/html5/thumbnails/57.jpg)
The$Kullback,Leibler$Divergence
![Page 58: Chris&Bishop,&PRML PATTERN&RECOGNITION& AND MACHINE&LEARNING€¦ · Chris&Bishop,&PRML PATTERN&RECOGNITION& ANDMACHINE&LEARNING CH&1&2:&INTROTO&PROBABILITY&DENSITY&ESTIMATION COMPSCI'591/691NR](https://reader033.vdocuments.net/reader033/viewer/2022042712/5f98cc31adc0b8014602511c/html5/thumbnails/58.jpg)
Mutual&Information