CS Statistical Machine learning Lecture 12 Yuan (Alan) Qi Purdue CS Oct Outline Review of Probit regression, Laplace approximation, BIC, Bayesian logistic regression Kernel methods Kernel ridge regression Kernel Principle Component Analysis Probit Regression Probit function: Labeling Noise Model Robust to outliers and labeling errors Generalized Linear Models Generalized linear model: Activation function: Link function: Canonical Link Function If we choose the canonical link function: Gradient of the error function: Examples Laplace Approximation for Posterior Gaussian approximation around mode: Evidence Approximation Bayesian Information Criterion Approximation of Laplace approximation: More accurate evidence approximation needed Bayesian Logistic Regression Kernel Methods Predictions are linear combinations of a kernel function evaluated at training data points. Kernel function feature space mapping Linear kernel: Stationary kernels: Fast Evaluation of Inner Product of Feature Mappings by Kernel Functions Inner product needs computing six feature values and 3 x 3 = 9 multiplications Kernel function has 2 multiplications and a squaring Kernel Trick 1. Reformulate an algorithm such that input vector enters only in the form of inner product. 2. Replace input x by its feature mapping: 3. Replace the inner product by a Kernel function: Examples: Kernel PCA, Kernel Fisher discriminant, Support Vector Machines Dual variables: Dual Representation for Ridge Regression Kernel Ridge Regression Using kernel trick: Minimize over dual variables: Generate Kernel Matrix Positive semidefinite Consider Gaussian kernel: Combining Generative & Discriminative Models by Kernels Since each modeling approach has distinct advantages, how to combine them? Use generative models to construct kernels, Use these kernels in discriminative approaches Measure Probability Similarity by Kernels Simple inner product: For mixture distribution: For infinite mixture models: For models with latent variables (e.g,. Hidden Markov Models:) Fisher Kernels Fisher Score: Fisher Information Matrix: Fisher Kernel: Sample Average : Principle Component Analysis (PCA) Assume We have is a normalized eigenvector: Feature Mapping Eigen-problem in feature space Dual Variables Suppose, we have Eigen-problem in Feature Space (1) Eigen-problem in Feature Space (2) Normalization condition: Projection coefficient: General Case for Non-zero Mean Case Kernel Matrix: Kernel PCA on Synthetic Data Contour plots of projection coefficients in feature space Limitations of Kernel PCA Discussion