Restricted Boltzmann Machines and Restricted Boltzmann Machines and Deep Networks for Unsupervised LearningDeep Networks for Unsupervised Learning

Restricted Boltzmann Machines and Restricted Boltzmann Machines and Deep Networks for Unsupervised LearningDeep Networks for Unsupervised Learning

Instituto Italiano di Tecnologia, Genova June 7th, 2011

Loris BazzaniUniversity of Verona

Brief Intro

• Unsupervised Learning

• Learning features from (visual) data

• Focus here on Restricted Boltzmann Machines

2

Outline Presentation

• Restricted Boltzmann Machines (RBMs)– Binary RBMs– Gaussian-binary RBMs– RBMs for Classification– Deep Belief Networks (DBNs)

• Learning Algorithms

Appl

icati

ons

Theo

ry

• RBMs for Modeling Natural Scenes [Ranzato, CVPR 2010]

• Learning Attentional Policies [Bazzani, ICML 2011]

Outline Presentation

• Restricted Boltzmann Machines (RBMs)– Binary RBMs– Gaussian-binary RBMs– RBMs for Classification– Deep Belief Networks (DBNs)

• Learning Algorithms

Appl

icati

ons

Theo

ry

• RBMs for Modeling Natural Scenes [Ranzato, CVPR 2010]

• Learning Attentional Policies [Bazzani, ICML 2011]

Restricted Boltzmann Machines

5

• Bipartite Probabilistic Graphical Model

W: parameters governing the interactions between visible and hidden units

• Property: ”given the hidden units, all of the visible units become independent and given the visible units, all of the hidden units become independent”

Binary RBMs

• We can sample from

• Use the expected value of the hidden units as features:

6

Gaussian-binary RBMs

• Popular extension for modeling natural images

• Make the visible units conditionally Gaussian given the hidden units

• Conditional distributions

7

RBMs for Classification

1) Feed the hidden representation into a standard classifier (e.g., multinomial logistic regression, SVM, random forest,…)

2) Embed the class into the visible units

and, the class vector will be sample from

8

Deep Belief Networks

• Goal: reach a high level of abstraction, so that classification becomes simple (e.g., linear)

• Multiple stacked RBMs• Learning consists in greedy training each

level sequentially from the bottom• Add fine-tuning with back-propagation• Or a non-linear classifier can be used• PB: how many layers?

9

Outline Presentation

• Restricted Boltzmann Machines (RBMs)– Binary RBMs– Gaussian-binary RBMs– RBMs for Classification– Deep Belief Networks (DBNs)

• Learning Algorithms

Appl

icati

ons

Theo

ry

• RBMs for Modeling Natural Scenes [Ranzato, CVPR 2010]

• Learning Attentional Policies [Bazzani, ICML 2011]

• Maximum Likelihood (ML) techniques• No close-form solution for the maximization• Problem: Partition function usually not

efficiently computable• Solutions:

• Approximate ML• Sacrifices convergence properties to make it

computationally feasible• Alternatives: variational methods, max-margin learning, etc.

How to Learn the Parameters

11

ML Problem

12

Gradient:

Match the gradient of the free energy under the data distribution with the gradient under the model distribution

Match the gradient of the free energy under the data distribution with the gradient under the model distribution

Marginalizing

Contrastive Divergence (1)

• It is just a gradient descent• At each step, it contrasts the data distribution

with the model distribution

• E.g., binary RBM

13

Contrastive Divergence (2)

• Algorithm for binary RBMs:

14

Outline Presentation

• Restricted Boltzmann Machines (RBMs)– Binary RBMs– Gaussian-binary RBMs– RBMs for Classification– Deep Belief Networks (DBNs)

• Learning Algorithms

Appl

icati

ons

Theo

ry

• RBMs for Modeling Natural Scenes [Ranzato, CVPR 2010]

• Learning Attentional Policies [Bazzani, ICML 2011]

Modeling Natural Images

• Motivations:– Learning a generative model of natural images– Extracting features that capture regularities– Opposed to using engineered features

• RBM with two set of hidden units:– One represents the pixel intensity– Another one, the pair-wise dependencies

• Called Mean-Covariance RBM (mc-RBM)• It is still a Gaussian-binary RBM

17

mc-RBM Model (1)

• Capture pair-wire interactions with:

• Sketch:

18

Covariancehiddens

mc-RBM Model (2)

• Representation of mean pixel intensities:

• Conditional distributions:

19

mc-RBM Model (3)

• Final Energy term:

• Free Energy formulation is also computable• Learning with

– Stochastic gradient descent– And, Contrastive Divergence– Sampling using Hybrid Monte Carlo

20

Regularization

Training Protocol for Recognition

• Images are pre-processed by PCA whitening• Train the mc-RBM• Extract features with mc-RBM• Train a classifier for object recognition:– Multinomial Logistic Classifier

21

Object Recognition on CIFAR 10

22

Outline Presentation

• Restricted Boltzmann Machines (RBMs)– Binary RBMs– Gaussian-binary RBMs– RBMs for Classification– Deep Belief Networks (DBNs)

• Learning Algorithms

Appl

icati

ons

Theo

ry

• RBMs for Modeling Natural Scenes [Ranzato, CVPR 2010]

• Learning Attentional Policies [Bazzani, ICML 2011]

Where do you look at?Where do you look at?

Original video source: http://gpu4vision.icg.tugraz.at/index.php?content=subsites/prost/prost.php

Goal

• Human tracking and recognition is amazingly efficient and effective

• Large stream of data is filtered by attention• We propose a model for tracking and recognition

that takes inspiration from human visual system• Tracking and recognition of “something” that is

moving in the scene• Accumulate gaze data• Plan where to look at in the next future

25

Parallelism with Human Brain

26Source image: http://www.waece.org/cd_morelia2006/ponencias/stoodley.htm

Sketch of the Model

27

(mc-)RBM(mc-)RBM

Multi-fixation RBMMulti-fixation RBM

ClassifierClassifier

Policy LearningPolicy Learning

Learning

• Offline Training• Extract gaze data from a training dataset

• Train the (mc-)RBM

• Train the multi-fixation RBM (3 random gazes)

• Train the multinomial logistic classifier

28

• Online Learning

• Hedge algorithm for policy learning

Online

from moving “things” with multiple saccades

Modularity

autoencoders, sparse coding, etc.

SVM, random forest, etc.

other bandit techniques or Bayesian optimization

• 10 synthetic video sequences with moving and background digits (from MNIST dataset)

Experiments (1)

Tracking error in pixels

Classification accuracy

Code available at: http://www.lorisbazzani.info/code-datasets/rbm-tracking/ Code available at: http://www.lorisbazzani.info/code-datasets/rbm-tracking/

Experiments (2)

Dataset available at: http://seqam.rutgers.edu/softdata/facedata/facedata.html

Summary

• Several RBMs models• How to train RBMs• Their extensions for classification• RBMs as block for deep architectures• They are useful for learning features from

images, without engineering them• Taking inspiration from human learning, DBNs

have been used

32

References (1)Learning attentional policies for tracking and recognition in video with deep networks, Loris Bazzani,

Nando de Freitas, Hugo Larochelle, Vittorio Murino, and Jo-Anne Ting, International Conference on Machine Learning, 2011

Tutorial on Stochastic Approximation Algorithms for Training Restricted Boltzmann Machines and Deep Belief Nets, Swersky and Bo Chen, Benjamin Marlin, and Nando de Freitas, Information Theory and Applications (ITA) Workshop, 2010

Inductive Principles for Restricted Boltzmann Machine Learning, Benjamin Marlin, Kevin Swersky, Bo Chen, and Nando de Freitas, AISTATS, 2010

Modeling Pixel Means and Covariances Using Factorized Third-Order Boltzmann, Marc'Aurelio Ranzato and Geoffrey E. Hinton, IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2010

Factored 3-Way Restricted Boltzmann Machines For Modeling Natural Images, Marc'Aurelio Ranzato, Alex Krizhevsky and Geoffrey E. Hinton, International Conference on Artificial Intelligence and Statistics, 2010

On Deep Generative Models with Applications to Recognition, Marc'Aurelio Ranzato, Joshua Susskind, Volodymyr Mnih, and Geoffrey Hinton, IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2011

Learning to combine foveal glimpses with a third-order Boltzmann machine, Hugo Larochelle and Geoffrey E. Hinton, Neural Information Processing Systems, 2010

33

References (2)Stacks of Convolutional Restricted Boltzmann Machines for Shift-Invariant Feature Learning,

Mohammad Norouzi, Mani Ranjbar, and Greg Mori, IEEE Computer Society Conference on Computer Vision and Pattern Recognition,2009

Deconvolutional Networks, Matthew D. Zeiler, Dilip Krishnan, Graham W. Taylor, and Rob Fergus, IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2010

Convolutional deep belief networks for scalable unsupervised learning of hierarchical representations, Lee, Honglak, Grosse, Roger, Ranganath, Rajesh and Ng, Andrew, International Conference on Machine Learning, 2009

A deep learning approach to machine transliteration, Deselaers, Thomas, Hasan, Savsa, Bender, Oliver and Ney, Hermann, Proceedings of the Fourth Workshop on Statistical Machine Translation, 2009

Learning Multilevel Distributed Representations for High-dimensional Sequences, Sutskever, I. and Hinton, G. E., Proceeding of the Eleventh International Conference on Artificial Intelligence and Statistics, 2007

On Contrastive Divergence Learning, Miguel A. Carreira-Perpinan and Geoffrey E. Hinton, International Conference on Artificial Intelligence and Statistics, 2005

Convolutional learning of spatio-temporal features, Taylor, Graham W., Fergus, Rob, LeCun, Yann and Bregler, Christoph, Proceedings of the 11th European conference on Computer vision, 2010

A Practical Guide to Training Restricted Boltzmann Machines, Geoffrey E. Hinton, University of Toronto, 2010, TR2010-003

34