eecs 442 computer vision optical flow and tracking 442 – computer vision optical flow and tracking...
TRANSCRIPT
![Page 1: EECS 442 Computer vision Optical flow and tracking 442 – Computer vision Optical flow and tracking • Intro • Optical flow and feature tracking • Lucas-Kanade algorithm •](https://reader031.vdocuments.net/reader031/viewer/2022021510/5aaf45377f8b9a07498d1a05/html5/thumbnails/1.jpg)
EECS 442 – Computer vision
Optical flow and tracking
• Intro
• Optical flow and feature tracking
• Lucas-Kanade algorithm
• Motion segmentation
Segments of this lectures are courtesy of Profs S. Lazebnik
S. Seitz, R. Szeliski, M. Pollefeys, K. Hassan-Shafique. S. Thrun
![Page 2: EECS 442 Computer vision Optical flow and tracking 442 – Computer vision Optical flow and tracking • Intro • Optical flow and feature tracking • Lucas-Kanade algorithm •](https://reader031.vdocuments.net/reader031/viewer/2022021510/5aaf45377f8b9a07498d1a05/html5/thumbnails/2.jpg)
From images to videos
• A video is a sequence of frames captured over time
• Now our image data is a function of space
(x, y) and time (t)
![Page 3: EECS 442 Computer vision Optical flow and tracking 442 – Computer vision Optical flow and tracking • Intro • Optical flow and feature tracking • Lucas-Kanade algorithm •](https://reader031.vdocuments.net/reader031/viewer/2022021510/5aaf45377f8b9a07498d1a05/html5/thumbnails/3.jpg)
Uses of motion
• Tracking features
• Segmenting objects based on motion cues
• Tracking objects
• Recognizing events and activities
• Improving video quality
– motion stabilization
– Super resolution
![Page 4: EECS 442 Computer vision Optical flow and tracking 442 – Computer vision Optical flow and tracking • Intro • Optical flow and feature tracking • Lucas-Kanade algorithm •](https://reader031.vdocuments.net/reader031/viewer/2022021510/5aaf45377f8b9a07498d1a05/html5/thumbnails/4.jpg)
Estimating 3D structure by tracking
Courtesy of Jean-Yves Bouguet – Vision Lab, California Institute of Technology
![Page 5: EECS 442 Computer vision Optical flow and tracking 442 – Computer vision Optical flow and tracking • Intro • Optical flow and feature tracking • Lucas-Kanade algorithm •](https://reader031.vdocuments.net/reader031/viewer/2022021510/5aaf45377f8b9a07498d1a05/html5/thumbnails/5.jpg)
Segmenting objects based on
motion cues
• Background subtraction
– A static camera is observing a scene
– Goal: separate the static background from the moving foreground
![Page 6: EECS 442 Computer vision Optical flow and tracking 442 – Computer vision Optical flow and tracking • Intro • Optical flow and feature tracking • Lucas-Kanade algorithm •](https://reader031.vdocuments.net/reader031/viewer/2022021510/5aaf45377f8b9a07498d1a05/html5/thumbnails/6.jpg)
• Motion segmentation
– Segment the video into multiple coherently moving objects
S. J. Pundlik and S. T. Birchfield, Motion Segmentation at Any Speed,
Proceedings of the British Machine Vision Conference 06
Segmenting objects based on
motion cues
![Page 7: EECS 442 Computer vision Optical flow and tracking 442 – Computer vision Optical flow and tracking • Intro • Optical flow and feature tracking • Lucas-Kanade algorithm •](https://reader031.vdocuments.net/reader031/viewer/2022021510/5aaf45377f8b9a07498d1a05/html5/thumbnails/7.jpg)
Tracking objects
• Facing tracking on openCV
http://www.youtube.com/watch?v=HTk_UwAYzVk
OpenCV's face tracker uses an algorithm called Camshift (based on the meanshift
algorithm)
![Page 8: EECS 442 Computer vision Optical flow and tracking 442 – Computer vision Optical flow and tracking • Intro • Optical flow and feature tracking • Lucas-Kanade algorithm •](https://reader031.vdocuments.net/reader031/viewer/2022021510/5aaf45377f8b9a07498d1a05/html5/thumbnails/8.jpg)
Z.Yin and R.Collins, "On-the-fly Object Modeling while Tracking," IEEE Computer Vision and
Pattern Recognition (CVPR '07), Minneapolis, MN, June 2007, 8 pages.
Tracking objects
![Page 9: EECS 442 Computer vision Optical flow and tracking 442 – Computer vision Optical flow and tracking • Intro • Optical flow and feature tracking • Lucas-Kanade algorithm •](https://reader031.vdocuments.net/reader031/viewer/2022021510/5aaf45377f8b9a07498d1a05/html5/thumbnails/9.jpg)
Joint tracking and 3D localization
W. Choi & K. Shahid & S. Savarese WMC 2009
W. Choi & S. Savarese , ECCV, 2010
![Page 10: EECS 442 Computer vision Optical flow and tracking 442 – Computer vision Optical flow and tracking • Intro • Optical flow and feature tracking • Lucas-Kanade algorithm •](https://reader031.vdocuments.net/reader031/viewer/2022021510/5aaf45377f8b9a07498d1a05/html5/thumbnails/10.jpg)
Joint tracking and 3D localization
W. Choi & K. Shahid & S. Savarese WMC 2009
W. Choi & S. Savarese , ECCV, 2010
![Page 11: EECS 442 Computer vision Optical flow and tracking 442 – Computer vision Optical flow and tracking • Intro • Optical flow and feature tracking • Lucas-Kanade algorithm •](https://reader031.vdocuments.net/reader031/viewer/2022021510/5aaf45377f8b9a07498d1a05/html5/thumbnails/11.jpg)
Tracking body parts
Courtesy of Benjamin Sapp
![Page 12: EECS 442 Computer vision Optical flow and tracking 442 – Computer vision Optical flow and tracking • Intro • Optical flow and feature tracking • Lucas-Kanade algorithm •](https://reader031.vdocuments.net/reader031/viewer/2022021510/5aaf45377f8b9a07498d1a05/html5/thumbnails/12.jpg)
D. Ramanan, D. Forsyth, and A. Zisserman. Tracking People by Learning their Appearance. PAMI 2007.
Tracker
Recognizing events and activities
![Page 13: EECS 442 Computer vision Optical flow and tracking 442 – Computer vision Optical flow and tracking • Intro • Optical flow and feature tracking • Lucas-Kanade algorithm •](https://reader031.vdocuments.net/reader031/viewer/2022021510/5aaf45377f8b9a07498d1a05/html5/thumbnails/13.jpg)
Juan Carlos Niebles, Hongcheng Wang and Li Fei-Fei, Unsupervised Learning of Human Action
Categories Using Spatial-Temporal Words, (BMVC), Edinburgh, 2006.
Recognizing events and activities
![Page 14: EECS 442 Computer vision Optical flow and tracking 442 – Computer vision Optical flow and tracking • Intro • Optical flow and feature tracking • Lucas-Kanade algorithm •](https://reader031.vdocuments.net/reader031/viewer/2022021510/5aaf45377f8b9a07498d1a05/html5/thumbnails/14.jpg)
Crossing – Talking – Queuing – Dancing – jogging
W. Choi & K. Shahid & S. Savarese WMC 2010
Recognizing group activities
![Page 16: EECS 442 Computer vision Optical flow and tracking 442 – Computer vision Optical flow and tracking • Intro • Optical flow and feature tracking • Lucas-Kanade algorithm •](https://reader031.vdocuments.net/reader031/viewer/2022021510/5aaf45377f8b9a07498d1a05/html5/thumbnails/16.jpg)
16
Super-resolution
Example: A set of low
quality images
![Page 17: EECS 442 Computer vision Optical flow and tracking 442 – Computer vision Optical flow and tracking • Intro • Optical flow and feature tracking • Lucas-Kanade algorithm •](https://reader031.vdocuments.net/reader031/viewer/2022021510/5aaf45377f8b9a07498d1a05/html5/thumbnails/17.jpg)
17
Super-resolution
Each of these images
looks like this:
![Page 18: EECS 442 Computer vision Optical flow and tracking 442 – Computer vision Optical flow and tracking • Intro • Optical flow and feature tracking • Lucas-Kanade algorithm •](https://reader031.vdocuments.net/reader031/viewer/2022021510/5aaf45377f8b9a07498d1a05/html5/thumbnails/18.jpg)
18
Super-resolution
The recovery result:
![Page 19: EECS 442 Computer vision Optical flow and tracking 442 – Computer vision Optical flow and tracking • Intro • Optical flow and feature tracking • Lucas-Kanade algorithm •](https://reader031.vdocuments.net/reader031/viewer/2022021510/5aaf45377f8b9a07498d1a05/html5/thumbnails/19.jpg)
Motion estimation techniques
• Optical flow – Recover image motion at each pixel from spatio-temporal
image brightness variations (optical flow)
• Feature-tracking – Extract visual features (corners, textured areas) and
“track” them over multiple frames
![Page 20: EECS 442 Computer vision Optical flow and tracking 442 – Computer vision Optical flow and tracking • Intro • Optical flow and feature tracking • Lucas-Kanade algorithm •](https://reader031.vdocuments.net/reader031/viewer/2022021510/5aaf45377f8b9a07498d1a05/html5/thumbnails/20.jpg)
Picture courtesy of Selim Temizer - Learning and Intelligent Systems (LIS) Group, MIT
Optical flow
Vector field function of the
spatio-temporal image
brightness variations
http://www.youtube.com/watch?v=JlLkkom
6tWw
![Page 21: EECS 442 Computer vision Optical flow and tracking 442 – Computer vision Optical flow and tracking • Intro • Optical flow and feature tracking • Lucas-Kanade algorithm •](https://reader031.vdocuments.net/reader031/viewer/2022021510/5aaf45377f8b9a07498d1a05/html5/thumbnails/21.jpg)
Feature-tracking
Courtesy of Jean-Yves Bouguet – Vision Lab, California Institute of Technology
![Page 22: EECS 442 Computer vision Optical flow and tracking 442 – Computer vision Optical flow and tracking • Intro • Optical flow and feature tracking • Lucas-Kanade algorithm •](https://reader031.vdocuments.net/reader031/viewer/2022021510/5aaf45377f8b9a07498d1a05/html5/thumbnails/22.jpg)
Optical flow
Definition: optical flow is the apparent motion of
brightness patterns in the image
Note: apparent motion can be caused by
lighting changes without any actual motion • Think of a uniform rotating sphere under fixed lighting vs. a
stationary sphere under moving illumination
GOAL: Recover image motion at each pixel by
optical flow
![Page 23: EECS 442 Computer vision Optical flow and tracking 442 – Computer vision Optical flow and tracking • Intro • Optical flow and feature tracking • Lucas-Kanade algorithm •](https://reader031.vdocuments.net/reader031/viewer/2022021510/5aaf45377f8b9a07498d1a05/html5/thumbnails/23.jpg)
Estimating optical flow
Given two subsequent frames, estimate the apparent motion
field u(x,y), v(x,y) between them
• Key assumptions • Brightness constancy: projection of the same point looks the
same in every frame
• Small motion: points do not move very far
• Spatial coherence: points move like their neighbors
I(x,y,t–1) I(x,y,t)
![Page 24: EECS 442 Computer vision Optical flow and tracking 442 – Computer vision Optical flow and tracking • Intro • Optical flow and feature tracking • Lucas-Kanade algorithm •](https://reader031.vdocuments.net/reader031/viewer/2022021510/5aaf45377f8b9a07498d1a05/html5/thumbnails/24.jpg)
tyx IyxvIyxuItyxItuyuxI ),(),()1,,(),,(
Brightness Constancy Equation:
),()1,,( ),,(),( tyxyx vyuxItyxI
Linearizing the right side using Taylor expansion:
The brightness constancy constraint
I(x,y,t–1) I(x,y,t)
0 tyx IvIuIHence,
Image derivative along x
0IvuI t
T
tyx IyxvIyxuItyxItuyuxI ),(),()1,,(),,(
u
v
![Page 25: EECS 442 Computer vision Optical flow and tracking 442 – Computer vision Optical flow and tracking • Intro • Optical flow and feature tracking • Lucas-Kanade algorithm •](https://reader031.vdocuments.net/reader031/viewer/2022021510/5aaf45377f8b9a07498d1a05/html5/thumbnails/25.jpg)
The brightness constancy constraint
How many equations and unknowns per pixel?
•One equation (this is a scalar equation!), two unknowns (u,v)
0IvuI t
T
Can we use this equation to recover image motion (u,v) at
each pixel?
![Page 26: EECS 442 Computer vision Optical flow and tracking 442 – Computer vision Optical flow and tracking • Intro • Optical flow and feature tracking • Lucas-Kanade algorithm •](https://reader031.vdocuments.net/reader031/viewer/2022021510/5aaf45377f8b9a07498d1a05/html5/thumbnails/26.jpg)
Adding constraints….
How to get more equations for a pixel?
Spatial coherence constraint:
Assume the pixel’s neighbors have the same (u,v) • If we use a 5x5 window, that gives us 25 equations per pixel
B. Lucas and T. Kanade. An iterative image registration technique with an application to stereo vision. In
Proceedings of the International Joint Conference on Artificial Intelligence, pp. 674–679, 1981.
pi = (xi, yi)
![Page 27: EECS 442 Computer vision Optical flow and tracking 442 – Computer vision Optical flow and tracking • Intro • Optical flow and feature tracking • Lucas-Kanade algorithm •](https://reader031.vdocuments.net/reader031/viewer/2022021510/5aaf45377f8b9a07498d1a05/html5/thumbnails/27.jpg)
Overconstrained linear system:
Lucas-Kanade flow
![Page 28: EECS 442 Computer vision Optical flow and tracking 442 – Computer vision Optical flow and tracking • Intro • Optical flow and feature tracking • Lucas-Kanade algorithm •](https://reader031.vdocuments.net/reader031/viewer/2022021510/5aaf45377f8b9a07498d1a05/html5/thumbnails/28.jpg)
Lucas-Kanade flow
Overconstrained linear system
The summations are over all pixels in the K x K window
Least squares solution for d given by
![Page 29: EECS 442 Computer vision Optical flow and tracking 442 – Computer vision Optical flow and tracking • Intro • Optical flow and feature tracking • Lucas-Kanade algorithm •](https://reader031.vdocuments.net/reader031/viewer/2022021510/5aaf45377f8b9a07498d1a05/html5/thumbnails/29.jpg)
Conditions for solvability
• Optimal (u, v) satisfies Lucas-Kanade equation
Does this remind anything to you?
When is this Solvable? • ATA should be invertible
• ATA should not be too small due to noise
– eigenvalues 1 and 2 of ATA should not be too small
• ATA should be well-conditioned
– 1/ 2 should not be too large ( 1 = larger eigenvalue)
![Page 30: EECS 442 Computer vision Optical flow and tracking 442 – Computer vision Optical flow and tracking • Intro • Optical flow and feature tracking • Lucas-Kanade algorithm •](https://reader031.vdocuments.net/reader031/viewer/2022021510/5aaf45377f8b9a07498d1a05/html5/thumbnails/30.jpg)
• Eigenvectors and eigenvalues of ATA relate to
edge direction and magnitude • The eigenvector associated with the larger eigenvalue points
in the direction of fastest intensity change
• The other eigenvector is orthogonal to it
M = ATA is the second moment matrix !
(Harris corner detector…)
M =
![Page 31: EECS 442 Computer vision Optical flow and tracking 442 – Computer vision Optical flow and tracking • Intro • Optical flow and feature tracking • Lucas-Kanade algorithm •](https://reader031.vdocuments.net/reader031/viewer/2022021510/5aaf45377f8b9a07498d1a05/html5/thumbnails/31.jpg)
Interpreting the eigenvalues
1
2
“Corner”
1 and 2 are large,
1 ~ 2
1 and 2 are small “Edge”
1 >> 2
“Edge”
2 >> 1
“Flat”
region
Classification of image points using eigenvalues
of the second moment matrix:
![Page 32: EECS 442 Computer vision Optical flow and tracking 442 – Computer vision Optical flow and tracking • Intro • Optical flow and feature tracking • Lucas-Kanade algorithm •](https://reader031.vdocuments.net/reader031/viewer/2022021510/5aaf45377f8b9a07498d1a05/html5/thumbnails/32.jpg)
Edge
– gradients very large or very small
– large 1, small 2
![Page 33: EECS 442 Computer vision Optical flow and tracking 442 – Computer vision Optical flow and tracking • Intro • Optical flow and feature tracking • Lucas-Kanade algorithm •](https://reader031.vdocuments.net/reader031/viewer/2022021510/5aaf45377f8b9a07498d1a05/html5/thumbnails/33.jpg)
Low-texture region
– gradients have small magnitude
– small 1, small 2
![Page 34: EECS 442 Computer vision Optical flow and tracking 442 – Computer vision Optical flow and tracking • Intro • Optical flow and feature tracking • Lucas-Kanade algorithm •](https://reader031.vdocuments.net/reader031/viewer/2022021510/5aaf45377f8b9a07498d1a05/html5/thumbnails/34.jpg)
High-texture region
– gradients are different, large magnitudes
– large 1, large 2
![Page 35: EECS 442 Computer vision Optical flow and tracking 442 – Computer vision Optical flow and tracking • Intro • Optical flow and feature tracking • Lucas-Kanade algorithm •](https://reader031.vdocuments.net/reader031/viewer/2022021510/5aaf45377f8b9a07498d1a05/html5/thumbnails/35.jpg)
What are good features to track?
Can we measure “quality” of features from just a
single image
Good features to track: - Harris corners (guarantee small error sensitivity)
Bad features to track: - Image points when either 1 or 2 (or both) is small (i.e., edges or
uniform textured regions)
![Page 36: EECS 442 Computer vision Optical flow and tracking 442 – Computer vision Optical flow and tracking • Intro • Optical flow and feature tracking • Lucas-Kanade algorithm •](https://reader031.vdocuments.net/reader031/viewer/2022021510/5aaf45377f8b9a07498d1a05/html5/thumbnails/36.jpg)
(u’,v’)
Ambiguities in tracking a point on a line
The component of the flow perpendicular to the gradient
(i.e., parallel to the edge) cannot be measured
edge
gradient
This equation
is always satisfied when (u’, v’ ) is
perpendicular to the image
gradient
0'v'uIT
![Page 37: EECS 442 Computer vision Optical flow and tracking 442 – Computer vision Optical flow and tracking • Intro • Optical flow and feature tracking • Lucas-Kanade algorithm •](https://reader031.vdocuments.net/reader031/viewer/2022021510/5aaf45377f8b9a07498d1a05/html5/thumbnails/37.jpg)
The barber pole illusion
http://en.wikipedia.org/wiki/Barberpole_illusion
![Page 38: EECS 442 Computer vision Optical flow and tracking 442 – Computer vision Optical flow and tracking • Intro • Optical flow and feature tracking • Lucas-Kanade algorithm •](https://reader031.vdocuments.net/reader031/viewer/2022021510/5aaf45377f8b9a07498d1a05/html5/thumbnails/38.jpg)
The barber pole illusion
http://en.wikipedia.org/wiki/Barberpole_illusion
![Page 39: EECS 442 Computer vision Optical flow and tracking 442 – Computer vision Optical flow and tracking • Intro • Optical flow and feature tracking • Lucas-Kanade algorithm •](https://reader031.vdocuments.net/reader031/viewer/2022021510/5aaf45377f8b9a07498d1a05/html5/thumbnails/39.jpg)
40
* From Marc Pollefeys COMP 256 2003
Aperture problem cont’d
![Page 40: EECS 442 Computer vision Optical flow and tracking 442 – Computer vision Optical flow and tracking • Intro • Optical flow and feature tracking • Lucas-Kanade algorithm •](https://reader031.vdocuments.net/reader031/viewer/2022021510/5aaf45377f8b9a07498d1a05/html5/thumbnails/40.jpg)
Motion estimation techniques
Optical flow • Recover image motion at each pixel from spatio-temporal
image brightness variations (optical flow)
Feature-tracking • Extract visual features (corners, textured areas) and
“track” them over multiple frames
• Shi-Tomasi feature tracker
• Tracking with dynamics
• Implemented in Open CV
![Page 41: EECS 442 Computer vision Optical flow and tracking 442 – Computer vision Optical flow and tracking • Intro • Optical flow and feature tracking • Lucas-Kanade algorithm •](https://reader031.vdocuments.net/reader031/viewer/2022021510/5aaf45377f8b9a07498d1a05/html5/thumbnails/41.jpg)
Shi-Tomasi feature tracker
Find good features using eigenvalues of second-
moment matrix • Key idea: “good” features to track are the ones that can be
tracked reliably
From frame to frame, track with Lucas-Kanade and a
pure translation model • More robust for small displacements, can be estimated from
smaller neighborhoods
Check consistency of tracks by affine registration to the
first observed instance of the feature • Affine model is more accurate for larger displacements
• Comparing to the first frame helps to minimize drift
J. Shi and C. Tomasi. Good Features to Track. CVPR 1994.
![Page 42: EECS 442 Computer vision Optical flow and tracking 442 – Computer vision Optical flow and tracking • Intro • Optical flow and feature tracking • Lucas-Kanade algorithm •](https://reader031.vdocuments.net/reader031/viewer/2022021510/5aaf45377f8b9a07498d1a05/html5/thumbnails/42.jpg)
Tracking example
![Page 43: EECS 442 Computer vision Optical flow and tracking 442 – Computer vision Optical flow and tracking • Intro • Optical flow and feature tracking • Lucas-Kanade algorithm •](https://reader031.vdocuments.net/reader031/viewer/2022021510/5aaf45377f8b9a07498d1a05/html5/thumbnails/43.jpg)
• Key assumptions (Errors in Lucas-Kanade)
• Small motion: points do not move very far
• Brightness constancy: projection of the same point
looks the same in every frame
• Spatial coherence: points move like their neighbors
Recap
![Page 44: EECS 442 Computer vision Optical flow and tracking 442 – Computer vision Optical flow and tracking • Intro • Optical flow and feature tracking • Lucas-Kanade algorithm •](https://reader031.vdocuments.net/reader031/viewer/2022021510/5aaf45377f8b9a07498d1a05/html5/thumbnails/44.jpg)
Revisiting the small motion assumption
Is this motion small enough? • Probably not—it’s much larger than one pixel (2nd order terms dominate)
• How might we solve this problem?
* From Khurram Hassan-Shafique CAP5415 Computer Vision 2003
![Page 45: EECS 442 Computer vision Optical flow and tracking 442 – Computer vision Optical flow and tracking • Intro • Optical flow and feature tracking • Lucas-Kanade algorithm •](https://reader031.vdocuments.net/reader031/viewer/2022021510/5aaf45377f8b9a07498d1a05/html5/thumbnails/45.jpg)
Reduce the resolution!
* From Khurram Hassan-Shafique CAP5415 Computer Vision 2003
![Page 46: EECS 442 Computer vision Optical flow and tracking 442 – Computer vision Optical flow and tracking • Intro • Optical flow and feature tracking • Lucas-Kanade algorithm •](https://reader031.vdocuments.net/reader031/viewer/2022021510/5aaf45377f8b9a07498d1a05/html5/thumbnails/46.jpg)
Multi-resolution Lucas Kanade Algorithm
![Page 47: EECS 442 Computer vision Optical flow and tracking 442 – Computer vision Optical flow and tracking • Intro • Optical flow and feature tracking • Lucas-Kanade algorithm •](https://reader031.vdocuments.net/reader031/viewer/2022021510/5aaf45377f8b9a07498d1a05/html5/thumbnails/47.jpg)
image I image H
Gaussian pyramid of image 1 (t)
Gaussian pyramid of image 2 (t+1)
image 2 image 1 u=10 pixels
u=5 pixels
u=2.5 pixels
u=1.25 pixels
Coarse-to-fine optical flow estimation
![Page 48: EECS 442 Computer vision Optical flow and tracking 442 – Computer vision Optical flow and tracking • Intro • Optical flow and feature tracking • Lucas-Kanade algorithm •](https://reader031.vdocuments.net/reader031/viewer/2022021510/5aaf45377f8b9a07498d1a05/html5/thumbnails/48.jpg)
image I image J
Gaussian pyramid of image 1 (t) Gaussian pyramid of image 2 (t+1)
image 2 image 1
Coarse-to-fine optical flow estimation
run L-K
run L-K
warp & upsample
.
.
.
![Page 49: EECS 442 Computer vision Optical flow and tracking 442 – Computer vision Optical flow and tracking • Intro • Optical flow and feature tracking • Lucas-Kanade algorithm •](https://reader031.vdocuments.net/reader031/viewer/2022021510/5aaf45377f8b9a07498d1a05/html5/thumbnails/49.jpg)
Optical Flow Results
* From Khurram Hassan-Shafique CAP5415 Computer Vision 2003
![Page 50: EECS 442 Computer vision Optical flow and tracking 442 – Computer vision Optical flow and tracking • Intro • Optical flow and feature tracking • Lucas-Kanade algorithm •](https://reader031.vdocuments.net/reader031/viewer/2022021510/5aaf45377f8b9a07498d1a05/html5/thumbnails/50.jpg)
Optical Flow Results
* From Khurram Hassan-Shafique CAP5415 Computer Vision 2003
• http://www.ces.clemson.edu/~stb/klt/
• OpenCV
![Page 51: EECS 442 Computer vision Optical flow and tracking 442 – Computer vision Optical flow and tracking • Intro • Optical flow and feature tracking • Lucas-Kanade algorithm •](https://reader031.vdocuments.net/reader031/viewer/2022021510/5aaf45377f8b9a07498d1a05/html5/thumbnails/51.jpg)
• Key assumptions (Errors in Lucas-Kanade)
• Small motion: points do not move very far
• Brightness constancy: projection of the same point
looks the same in every frame
• Spatial coherence: points move like their neighbors
Recap
![Page 52: EECS 442 Computer vision Optical flow and tracking 442 – Computer vision Optical flow and tracking • Intro • Optical flow and feature tracking • Lucas-Kanade algorithm •](https://reader031.vdocuments.net/reader031/viewer/2022021510/5aaf45377f8b9a07498d1a05/html5/thumbnails/52.jpg)
EECS 442 – Computer vision
Optical flow and tracking
• Intro
• Optical flow and feature tracking
• Lucas-Kanade algorithm
• Motion segmentation
Segments of this lectures are courtesy of Profs S. Lazebnik
S. Seitz, R. Szeliski, M. Pollefeys, K. Hassan-Shafique. S. Thrun
![Page 53: EECS 442 Computer vision Optical flow and tracking 442 – Computer vision Optical flow and tracking • Intro • Optical flow and feature tracking • Lucas-Kanade algorithm •](https://reader031.vdocuments.net/reader031/viewer/2022021510/5aaf45377f8b9a07498d1a05/html5/thumbnails/53.jpg)
Motion segmentation
How do we represent the motion in this scene?
![Page 54: EECS 442 Computer vision Optical flow and tracking 442 – Computer vision Optical flow and tracking • Intro • Optical flow and feature tracking • Lucas-Kanade algorithm •](https://reader031.vdocuments.net/reader031/viewer/2022021510/5aaf45377f8b9a07498d1a05/html5/thumbnails/54.jpg)
Break image sequence into “layers” each of which has a
coherent (affine) motion
Motion segmentation J. Wang and E. Adelson. Layered Representation for Motion Analysis. CVPR 1993.
![Page 55: EECS 442 Computer vision Optical flow and tracking 442 – Computer vision Optical flow and tracking • Intro • Optical flow and feature tracking • Lucas-Kanade algorithm •](https://reader031.vdocuments.net/reader031/viewer/2022021510/5aaf45377f8b9a07498d1a05/html5/thumbnails/55.jpg)
Substituting into the brightness
constancy equation:
yaxaayxv
yaxaayxu
654
321
),(
),(
0 tyx IvIuI
Affine motion
![Page 56: EECS 442 Computer vision Optical flow and tracking 442 – Computer vision Optical flow and tracking • Intro • Optical flow and feature tracking • Lucas-Kanade algorithm •](https://reader031.vdocuments.net/reader031/viewer/2022021510/5aaf45377f8b9a07498d1a05/html5/thumbnails/56.jpg)
0)()( 654321 tyx IyaxaaIyaxaaI
Substituting into the brightness
constancy equation:
yaxaayxv
yaxaayxu
654
321
),(
),(
• Each pixel provides 1 linear constraint in
6 unknowns
2
tyx IyaxaaIyaxaaIaErr )()()( 654321
• If we have at least 6 pixels in a neighborhood,
a1… a6 can be found by least squares minimization:
Affine motion
![Page 57: EECS 442 Computer vision Optical flow and tracking 442 – Computer vision Optical flow and tracking • Intro • Optical flow and feature tracking • Lucas-Kanade algorithm •](https://reader031.vdocuments.net/reader031/viewer/2022021510/5aaf45377f8b9a07498d1a05/html5/thumbnails/57.jpg)
How do we estimate the layers?
1. Obtain a set of initial affine motion hypotheses
• Divide the image into blocks and estimate affine motion parameters in each
block by least squares
– Eliminate hypotheses with high residual error
2. Map into motion parameter space
3. Perform k-means clustering on affine motion parameters
–Merge clusters that are close and retain the largest clusters to obtain
a smaller set of hypotheses to describe all the motions in the scene
a1
a6
a2
![Page 58: EECS 442 Computer vision Optical flow and tracking 442 – Computer vision Optical flow and tracking • Intro • Optical flow and feature tracking • Lucas-Kanade algorithm •](https://reader031.vdocuments.net/reader031/viewer/2022021510/5aaf45377f8b9a07498d1a05/html5/thumbnails/58.jpg)
How do we estimate the layers?
1. Obtain a set of initial affine motion hypotheses
• Divide the image into blocks and estimate affine motion parameters in each
block by least squares
– Eliminate hypotheses with high residual error
2. Map into motion parameter space
3. Perform k-means clustering on affine motion parameters
–Merge clusters that are close and retain the largest clusters to obtain
a smaller set of hypotheses to describe all the motions in the scene
4. Assign each pixel to best hypothesis --- iterate
![Page 59: EECS 442 Computer vision Optical flow and tracking 442 – Computer vision Optical flow and tracking • Intro • Optical flow and feature tracking • Lucas-Kanade algorithm •](https://reader031.vdocuments.net/reader031/viewer/2022021510/5aaf45377f8b9a07498d1a05/html5/thumbnails/59.jpg)
Example result
J. Wang and E. Adelson. Layered Representation for Motion Analysis. CVPR 1993.
![Page 60: EECS 442 Computer vision Optical flow and tracking 442 – Computer vision Optical flow and tracking • Intro • Optical flow and feature tracking • Lucas-Kanade algorithm •](https://reader031.vdocuments.net/reader031/viewer/2022021510/5aaf45377f8b9a07498d1a05/html5/thumbnails/60.jpg)
2D Target tracking using Kalman filter in MATLAB
by AliReza KashaniPour
http://www.mathworks.com/matlabcentral/fileexchange/14243
![Page 61: EECS 442 Computer vision Optical flow and tracking 442 – Computer vision Optical flow and tracking • Intro • Optical flow and feature tracking • Lucas-Kanade algorithm •](https://reader031.vdocuments.net/reader031/viewer/2022021510/5aaf45377f8b9a07498d1a05/html5/thumbnails/61.jpg)
http://www.micc.unifi.it/pernici/#alien
DOWNLOAD
http://www.micc.unifi.it/pernici/
FaceHugger: The ALIEN Tracker
• Use Scale Invariant Feature Transform (SIFT) when applied
to (flat) objects
![Page 62: EECS 442 Computer vision Optical flow and tracking 442 – Computer vision Optical flow and tracking • Intro • Optical flow and feature tracking • Lucas-Kanade algorithm •](https://reader031.vdocuments.net/reader031/viewer/2022021510/5aaf45377f8b9a07498d1a05/html5/thumbnails/62.jpg)
Optical flow without motion!
![Page 63: EECS 442 Computer vision Optical flow and tracking 442 – Computer vision Optical flow and tracking • Intro • Optical flow and feature tracking • Lucas-Kanade algorithm •](https://reader031.vdocuments.net/reader031/viewer/2022021510/5aaf45377f8b9a07498d1a05/html5/thumbnails/63.jpg)
EECS 442 – Computer vision
Next lecture
Object Recognition - intro