MACHINE LEARNING: FOUNDATIONS COURSETAU – 2012APROF. YISHAY MANSOUR

o TextonBoost :Joint Appearance, Shape and Context Modeling for Multi-Class Object Recognition and Segmentation

J. Shotton*, J. Winn†, C. Rother†, and A. Criminisi†

o * University of Cambridgeo † Microsoft Research Ltd, Cambridge, UK

Yaniv BarMarch 2013

GOAL

o Simultaneous recognition and segmentation:Efficiently detect a large number of object classes and give a pixel-perfect segmentation of an image into these classes.

DATA AND CLASSES Original Paper: 3 DBs.

Main DB: MSRC 21. MSRC 21-Class Object

Recognition Database 591 hand-labelled images

Original main DB was updated to MSRC 23.

MSRC 23-Class Object Recognition Database 592 hand-labelled images

HIGH LEVEL APPROACH

High-level description of approach:Learn classifier based on relative texture locations for each class. Classification is then refined.

Given an image, for each pixel:

- Texture-Layout features are calculated

- A boosting classifier gives the probability of the pixel belonging to each class

- The discriminative model combines the boosting output with low-level color, location, and edge information; image receives final label.

Most important part of the model is the Shape/Context Potential – it is significant for object recognition and very rough segmentation results.

Other potential such as Edge and Color refine the segmentation results.

(a) Original image, (b) Shape, (c) (b)+edge, (d) (c)+color

TEXTURE LAYOUT FEATURES

For modeling object shape, appearance and context we use a New texton-based features.This feature (texton) compact and efficient characterisation of local texture.

o The task is to recognize surfaces made from different materials on the basis of their texture appearance.o Different materials show different texture appearance.Moreover, texture appearance of the same material changes dramatically due to different viewpoint/lighting settings (specularities, shadows, and occlusions).

WHAT ARE TEXTONS

Computing texton maps:

Texton mapColours Texton Indices

Input image

Clustering

Filter BankConvolve 17-D filter bank (composed of gaussians, dogs, logs) with all training images

Responses are clustered with K-means

Each pixel is assigned a texton number

CALCULATING TEXTURE-LAYOUT FEATURES

Capturing appearance:

How Texture-Layout features jointly model texture and layout:

LEARNING

Learning is done with Joint Boost algorithm – A version of Multi class gentle boost algorithm.

I’ve used both AdaBoost.M1 and AdaBoost.Mh (multiclass reduction to binary which is due to the fact that AdaBoosting is only for binary classification).

THE GOOD AND BAD

The Good: Provides reasonable recognition + segmentation for many classes. Also, combines several good ideas. Most of previous works didn’t tackle the problem as a whole – rather, problems were treated separately.

The Bad: Does not beat past work (in terms of quantitative recognition results) and a bit hacky.

CODE-SEQUENCE OF EXECUTION

1. imagesTextonization.m(extract efficient images characterization)

2. calcModelFeatures.m(calculate the appearance (shape) potential

context)

3. trainModel.m(build a classification model)

4. testModel.m(test the classification model with test data)

RESULTS