themes in computer vision carlo tomasi. applications autonomous cars, planes, missiles, robots,......

35
Themes in Computer Vision Carlo Tomasi

Upload: shanon-hunter

Post on 11-Jan-2016

218 views

Category:

Documents


1 download

TRANSCRIPT

Page 1: Themes in Computer Vision Carlo Tomasi. Applications autonomous cars, planes, missiles, robots,... space exploration aid to the blind, ASL recognition

Themes in Computer Vision

Carlo Tomasi

Page 2: Themes in Computer Vision Carlo Tomasi. Applications autonomous cars, planes, missiles, robots,... space exploration aid to the blind, ASL recognition

Applications

• autonomous cars, planes, missiles, robots, ...• space exploration• aid to the blind, ASL recognition• manufacturing,

quality control• surveillance, security• image retrieval• medical imaging• ...• perceptual input for

cognition

(CMU NavLab ‘90)

Page 3: Themes in Computer Vision Carlo Tomasi. Applications autonomous cars, planes, missiles, robots,... space exploration aid to the blind, ASL recognition

Vision is Effortless to Us

• driving a car

• threading a needle

• recognizing a distant, occluded object

• understanding (flat!) pictures

• perceive the mood of a painting

Page 4: Themes in Computer Vision Carlo Tomasi. Applications autonomous cars, planes, missiles, robots,... space exploration aid to the blind, ASL recognition

Technical Difficulties

• 512x512x3x30 ≈ 23.5MB/s was a problem 10 years ago

• technology just gotgood enough

• great opportunity!

Page 5: Themes in Computer Vision Carlo Tomasi. Applications autonomous cars, planes, missiles, robots,... space exploration aid to the blind, ASL recognition

Fundamental Challenges I

• 3D2D implies information loss

• sensitivity to errors

• need for models

graphics

vision

Page 6: Themes in Computer Vision Carlo Tomasi. Applications autonomous cars, planes, missiles, robots,... space exploration aid to the blind, ASL recognition

Reconstruction and Geometry

must use redundancy toaddress sensitivity to noise

Page 7: Themes in Computer Vision Carlo Tomasi. Applications autonomous cars, planes, missiles, robots,... space exploration aid to the blind, ASL recognition

Reconstruction Example

(Tomasi & Kanade ‘91)

Page 8: Themes in Computer Vision Carlo Tomasi. Applications autonomous cars, planes, missiles, robots,... space exploration aid to the blind, ASL recognition

Fundamental Challenges II

• Appearance changes with viewpoint, i.e., the same thing looks different• Geometric changes: surface slant depends on

viewpoint• Photometric changes: surface brightness and

color depend on viewpoint• Occlusions: what is hidden depends on

viewpoint

• Ambiguity: different things look similar• Correspondence is hard

Page 9: Themes in Computer Vision Carlo Tomasi. Applications autonomous cars, planes, missiles, robots,... space exploration aid to the blind, ASL recognition

Photometric and Geometric Change

Page 10: Themes in Computer Vision Carlo Tomasi. Applications autonomous cars, planes, missiles, robots,... space exploration aid to the blind, ASL recognition

Occlusion

?

Page 11: Themes in Computer Vision Carlo Tomasi. Applications autonomous cars, planes, missiles, robots,... space exploration aid to the blind, ASL recognition

Technicality: Motion Blur

Page 12: Themes in Computer Vision Carlo Tomasi. Applications autonomous cars, planes, missiles, robots,... space exploration aid to the blind, ASL recognition

Wrong Correspondence

Page 13: Themes in Computer Vision Carlo Tomasi. Applications autonomous cars, planes, missiles, robots,... space exploration aid to the blind, ASL recognition

Simple Images are Harder

(Birchfield and Tomasi ‘01)

Page 14: Themes in Computer Vision Carlo Tomasi. Applications autonomous cars, planes, missiles, robots,... space exploration aid to the blind, ASL recognition

Models

• must be insensitive to• viewing position

changes• lighting changes• object configuration

changes• occlusion• clutter

• must be sensitive to• object changes!

Page 15: Themes in Computer Vision Carlo Tomasi. Applications autonomous cars, planes, missiles, robots,... space exploration aid to the blind, ASL recognition

Low-Level Models are General

Model: surfaces are smooth, connected

(Marr and Poggio ‘80)

Page 16: Themes in Computer Vision Carlo Tomasi. Applications autonomous cars, planes, missiles, robots,... space exploration aid to the blind, ASL recognition

Higher-Level Models Work Better…

•… when they are right• (and much worse when they are wrong)

(Lin and Tomasi ‘01)

Page 17: Themes in Computer Vision Carlo Tomasi. Applications autonomous cars, planes, missiles, robots,... space exploration aid to the blind, ASL recognition

State of the Artle

ft in

put i

mag

eground truth disparity

our

resu

ltdisparity error

(Lin and Tomasi, 01)

Page 18: Themes in Computer Vision Carlo Tomasi. Applications autonomous cars, planes, missiles, robots,... space exploration aid to the blind, ASL recognition

Fundamental Challenges III• An old problem in the

new context of recognition:• Variation of appearance:

Objects change over time, with context, viewpoint, lighting, pose, expression,…

• Similarity: Different objects look similar

• [BTW, objects do not always appear in isolation…]

(US Army FERET Database)

Page 19: Themes in Computer Vision Carlo Tomasi. Applications autonomous cars, planes, missiles, robots,... space exploration aid to the blind, ASL recognition

Modeling Images as Points12

n

...

...

1

2

n

principal componentsform an approximate basisfor all the images in the set

... ... ... ... ... ... ... ...

Page 20: Themes in Computer Vision Carlo Tomasi. Applications autonomous cars, planes, missiles, robots,... space exploration aid to the blind, ASL recognition

Example: Eigenfaces

(Turk, Pentland ‘91; Murase-Nayar ‘93; many others)

........................

...

=

the projection of a new imageonto the eigenbasis isa compressed representationof that image

can use this to recognize faces,synthesize new images, ...

Page 21: Themes in Computer Vision Carlo Tomasi. Applications autonomous cars, planes, missiles, robots,... space exploration aid to the blind, ASL recognition

Fundamental Challenges IV:

“read my lips”

“run”• Variation, self-occlusion,occlusion, clutter, …

Motions can be complex

Page 22: Themes in Computer Vision Carlo Tomasi. Applications autonomous cars, planes, missiles, robots,... space exploration aid to the blind, ASL recognition

Simple Models Are Fast

(Birchfield ‘98)

a head is an ellipse with two colors,surrounded by strong intensity gradients

Page 23: Themes in Computer Vision Carlo Tomasi. Applications autonomous cars, planes, missiles, robots,... space exploration aid to the blind, ASL recognition

(Bregler ‘93)

2D Articulated Models for Tracking

Page 24: Themes in Computer Vision Carlo Tomasi. Applications autonomous cars, planes, missiles, robots,... space exploration aid to the blind, ASL recognition

3D Models are More Accurate…

•… when they are right• [BTW, why is she wearing a black shirt?]

(Isard & Blake ‘99)

Page 25: Themes in Computer Vision Carlo Tomasi. Applications autonomous cars, planes, missiles, robots,... space exploration aid to the blind, ASL recognition

Probabilistic Models Handle Uncertainty

• world state , observation (image) • prior P()

• colors change moderately (?)• arms move with limited acceleration (boxing?)• the height of a head can only change so much (dancing?)• contours are smooth and change smoothly• balls follow the laws of gravity• …

• sensor model P(|)• image motion can be measured only so well• motion blurs the image• noise corrupts pixel values• ...

Page 26: Themes in Computer Vision Carlo Tomasi. Applications autonomous cars, planes, missiles, robots,... space exploration aid to the blind, ASL recognition

Bayesian Tracking

• Bayes’ rule: P(|) P(|) P()

• what is the world state likely to be, given that we observed the image ?

(Isard & Blake ‘99)

Page 27: Themes in Computer Vision Carlo Tomasi. Applications autonomous cars, planes, missiles, robots,... space exploration aid to the blind, ASL recognition

Even Higher Models May Be Needed

[MY COMPUTER CAN UNDERSTAND SIGN] computer No(1(HandsIpsi 1 1 0 S Out Down, NeutralIpsi 0 0 0 S Out Down)( ,-) 0(" " 0 -1 " " ", " " " " " " ") (",-) 0(" " -1 0 " " ", " " " " " " ") (",-) 0(" " 0 1 " " ", " " " " " " ") (",-) 1(" " 1 0 " " ", " " " " " " ")) understand No(1(HandIn 0 0 0 X Out Contra,NeutralOut 0 0 0 D Up Contra)(-,-) "(" 1 " " " " ", " " " " " " "))signs No(1( 0 0 0 B Up Out, - - - - - - -) (-,-) "(" 1 0 0 " " ", - - - - - - -))can No(1(HandUp 0 0 0 Out Contra,NeutralOut 0 0 -1 B Out Up) (-,-) "(" " " " " " ", " " " 1 " " "))

(Richards & Tomasi ‘02)

Page 28: Themes in Computer Vision Carlo Tomasi. Applications autonomous cars, planes, missiles, robots,... space exploration aid to the blind, ASL recognition

Fundamental Challenge V:Images are Diverse

Page 29: Themes in Computer Vision Carlo Tomasi. Applications autonomous cars, planes, missiles, robots,... space exploration aid to the blind, ASL recognition

Previous Work in Image Retrieval

Hulton Deutsch

Page 30: Themes in Computer Vision Carlo Tomasi. Applications autonomous cars, planes, missiles, robots,... space exploration aid to the blind, ASL recognition

Color and Texture Models

orientation

scal

e

text

ure

Page 31: Themes in Computer Vision Carlo Tomasi. Applications autonomous cars, planes, missiles, robots,... space exploration aid to the blind, ASL recognition

Image Distances

(Rubner & Tomasi ‘97)

Page 32: Themes in Computer Vision Carlo Tomasi. Applications autonomous cars, planes, missiles, robots,... space exploration aid to the blind, ASL recognition

(Rubner & Tomasi ‘97)

Page 33: Themes in Computer Vision Carlo Tomasi. Applications autonomous cars, planes, missiles, robots,... space exploration aid to the blind, ASL recognition

Retrieval by Refinement - 1

(Rubner & Tomasi ‘97)

Page 34: Themes in Computer Vision Carlo Tomasi. Applications autonomous cars, planes, missiles, robots,... space exploration aid to the blind, ASL recognition

Retrieval by Refinement - 2

(Rubner & Tomasi ‘97)

Page 35: Themes in Computer Vision Carlo Tomasi. Applications autonomous cars, planes, missiles, robots,... space exploration aid to the blind, ASL recognition

Vision is AI Complete

• Vision is an inverse problem

• Strong models of the world are required

• Vision implies reasoning about the world

• Vision is AI