Lecture 9.3 Introduction to Machine Learning

Idar Dyrdal

Machine learning (Pattern recognition)

• Recognition of individuals (instance recognition) • Discrimination between classes (pattern recognition, classification)

Training Labels

Training Images

Classifier Training

Supervised learning

Image Features

Trained Classifier

2

Pattern recognition in practice

Working applications of Image Pattern recognition: • Reading license plates, postal codes, bar codes • Character recognition • Automatic diagnosis of medical samples • Fingerprint recognition • Face detection and recognition • …

3

Classification system

4

Feature extractor Classifier

Pine

Birch

Camera

Image features for object recognition

5

Feature vector and feature space

6

Training of classifiers

7

Learn a function to predict the class from the given features

Region 1

Region 2

Decision boundary

Classifiers and training methods

• Bayes classifier • Nearest-neighbors and K-nearest-neighbors • Parzen windows • Linear and higher order discriminant functions • Neural nets • Support Vector Machines (SVM) • …

8

Class conditional probability density functions

9

Bayesian decision theory

Overview

10

x

Density estimation

Parametric methods: • Assume a given shape of the density

function • Use the training set to estimate the

unknown parameters.

Non-parametric (distribution free) methods: • Point estimation of the density using

the training set directly • Parzen windows • Nearest neighbor estimation (leads

directly to the nearest-neighbor and k-nearest-neighbor classifiers).

11

Example – Gaussian distribution:

Parameter estimation

12

Discriminant functions

13

Estimate of the density in a given point:

From Bayes rule:

Example of a discriminant function: Decision rule: Choose the class with maximum discriminant function value.

Example - linear classifier

14

Uncorrelated features and common covariance matrices Linear decision boundaries

Dimensionality reduction - linear transformations

• PCA, ICA, LLE, Isomap • PCA (Principal Components Analysis)

is one of the most important techniques for dimensionality reduction

• It takes advantage of correlations between the features to produce the best possible lower dimensional representation of the data with respect to reconstruction error

• The eigenvectors of the lumped covariance matrix defines the new features in the transformed feature space.

15

Summary

Recognition: • Pattern classification • Training of classifiers (supervised learning) • Parametric and non-parametric methods • Discriminant functions • Dimensionality reduction

Read also: Szeliski 14.1

16