klt tracker & triangulation class 6 read shi and tomasi’s paper on good features to track
Post on 22-Dec-2015
220 views
TRANSCRIPT
![Page 1: KLT tracker & triangulation Class 6 Read Shi and Tomasi’s paper on good features to track](https://reader033.vdocuments.net/reader033/viewer/2022052701/56649d775503460f94a592fe/html5/thumbnails/1.jpg)
KLT tracker & triangulationClass 6
Read Shi and Tomasi’s paper on good features to trackhttp://www.unc.edu/courses/2004fall/comp/290/089/papers/shi-tomasi-good-features-cvpr1994.pdf
Optional: Lucas-Kanade 20 Years On http://www.ri.cmu.edu/projects/project_515.html
![Page 2: KLT tracker & triangulation Class 6 Read Shi and Tomasi’s paper on good features to track](https://reader033.vdocuments.net/reader033/viewer/2022052701/56649d775503460f94a592fe/html5/thumbnails/2.jpg)
Feature matching vs. tracking
Extract features independently and then match by comparing descriptors
Extract features in first images and then try to find same feature back in next view
What is a good feature?
Image-to-image correspondences are key to passive triangulation-based 3D reconstruction
![Page 3: KLT tracker & triangulation Class 6 Read Shi and Tomasi’s paper on good features to track](https://reader033.vdocuments.net/reader033/viewer/2022052701/56649d775503460f94a592fe/html5/thumbnails/3.jpg)
Feature point extraction• Approximate SSD for small displacement Δ
• Find points for which the following is maximum
maximize smallest eigenvalue of M
![Page 4: KLT tracker & triangulation Class 6 Read Shi and Tomasi’s paper on good features to track](https://reader033.vdocuments.net/reader033/viewer/2022052701/56649d775503460f94a592fe/html5/thumbnails/4.jpg)
SIFT features
• Scale-space DoG maxima
• Verify minimum contrast and “cornerness”• Orientation from dominant gradient
• Descriptor based on gradient distributions0 2
0 2
![Page 5: KLT tracker & triangulation Class 6 Read Shi and Tomasi’s paper on good features to track](https://reader033.vdocuments.net/reader033/viewer/2022052701/56649d775503460f94a592fe/html5/thumbnails/5.jpg)
Feature tracking
• Identify features and track them over video• Small difference between frames• potential large difference overall
• Standard approach: KLT (Kanade-Lukas-Tomasi)
![Page 6: KLT tracker & triangulation Class 6 Read Shi and Tomasi’s paper on good features to track](https://reader033.vdocuments.net/reader033/viewer/2022052701/56649d775503460f94a592fe/html5/thumbnails/6.jpg)
• Brightness constancy assumption
Intermezzo: optical flow
(small motion)
• 1D example
possibility for iterative refinement
![Page 7: KLT tracker & triangulation Class 6 Read Shi and Tomasi’s paper on good features to track](https://reader033.vdocuments.net/reader033/viewer/2022052701/56649d775503460f94a592fe/html5/thumbnails/7.jpg)
• Brightness constancy assumption
Intermezzo: optical flow
(small motion)
• 2D example
(2 unknowns)
(1 constraint)?
isophote I(t)=Iisophote I(t+1)=I
the “aperture” problem
![Page 8: KLT tracker & triangulation Class 6 Read Shi and Tomasi’s paper on good features to track](https://reader033.vdocuments.net/reader033/viewer/2022052701/56649d775503460f94a592fe/html5/thumbnails/8.jpg)
Intermezzo: optical flow• How to deal with aperture problem?
Assume neighbors have same displacement
(3 constraints if color gradients are different)
![Page 9: KLT tracker & triangulation Class 6 Read Shi and Tomasi’s paper on good features to track](https://reader033.vdocuments.net/reader033/viewer/2022052701/56649d775503460f94a592fe/html5/thumbnails/9.jpg)
Lucas-Kanade
Assume neighbors have same displacement
least-squares:
![Page 10: KLT tracker & triangulation Class 6 Read Shi and Tomasi’s paper on good features to track](https://reader033.vdocuments.net/reader033/viewer/2022052701/56649d775503460f94a592fe/html5/thumbnails/10.jpg)
• Compute translation assuming it is small
Alternative derivation
differentiate:
Affine is also possible, but a bit harder (6x6 in stead of 2x2)
![Page 11: KLT tracker & triangulation Class 6 Read Shi and Tomasi’s paper on good features to track](https://reader033.vdocuments.net/reader033/viewer/2022052701/56649d775503460f94a592fe/html5/thumbnails/11.jpg)
Revisiting the small motion assumption
• Is this motion small enough?• Probably not—it’s much larger than one pixel
(2nd order terms dominate)• How might we solve this problem?
* From Khurram Hassan-Shafique CAP5415 Computer Vision 2003
![Page 12: KLT tracker & triangulation Class 6 Read Shi and Tomasi’s paper on good features to track](https://reader033.vdocuments.net/reader033/viewer/2022052701/56649d775503460f94a592fe/html5/thumbnails/12.jpg)
Reduce the resolution!
* From Khurram Hassan-Shafique CAP5415 Computer Vision 2003
![Page 13: KLT tracker & triangulation Class 6 Read Shi and Tomasi’s paper on good features to track](https://reader033.vdocuments.net/reader033/viewer/2022052701/56649d775503460f94a592fe/html5/thumbnails/13.jpg)
image It-1 image I
Gaussian pyramid of image It-1 Gaussian pyramid of image I
image Iimage It-1u=10 pixels
u=5 pixels
u=2.5 pixels
u=1.25 pixels
Coarse-to-fine optical flow estimation
slides fromBradsky and Thrun
![Page 14: KLT tracker & triangulation Class 6 Read Shi and Tomasi’s paper on good features to track](https://reader033.vdocuments.net/reader033/viewer/2022052701/56649d775503460f94a592fe/html5/thumbnails/14.jpg)
image Iimage J
Gaussian pyramid of image It-1 Gaussian pyramid of image I
image Iimage It-1
Coarse-to-fine optical flow estimation
run iterative L-K
run iterative L-K
warp & upsample
.
.
.
slides fromBradsky and Thrun
![Page 15: KLT tracker & triangulation Class 6 Read Shi and Tomasi’s paper on good features to track](https://reader033.vdocuments.net/reader033/viewer/2022052701/56649d775503460f94a592fe/html5/thumbnails/15.jpg)
Good feature to track
• Tracking
• Use same window in feature selection as for tracking itself
maximize minimal eigenvalue of MStrategy:
• Look for strong well distributed features, typically few hundreds• initialize and then track, renew feature when too many are lost
![Page 16: KLT tracker & triangulation Class 6 Read Shi and Tomasi’s paper on good features to track](https://reader033.vdocuments.net/reader033/viewer/2022052701/56649d775503460f94a592fe/html5/thumbnails/16.jpg)
Example
Simple displacement is sufficient between consecutive frames, but not to compare to reference template
![Page 17: KLT tracker & triangulation Class 6 Read Shi and Tomasi’s paper on good features to track](https://reader033.vdocuments.net/reader033/viewer/2022052701/56649d775503460f94a592fe/html5/thumbnails/17.jpg)
Example
![Page 18: KLT tracker & triangulation Class 6 Read Shi and Tomasi’s paper on good features to track](https://reader033.vdocuments.net/reader033/viewer/2022052701/56649d775503460f94a592fe/html5/thumbnails/18.jpg)
Synthetic example
![Page 19: KLT tracker & triangulation Class 6 Read Shi and Tomasi’s paper on good features to track](https://reader033.vdocuments.net/reader033/viewer/2022052701/56649d775503460f94a592fe/html5/thumbnails/19.jpg)
Good features to keep tracking
Perform affine alignment between first and last frameStop tracking features with too large errors
![Page 20: KLT tracker & triangulation Class 6 Read Shi and Tomasi’s paper on good features to track](https://reader033.vdocuments.net/reader033/viewer/2022052701/56649d775503460f94a592fe/html5/thumbnails/20.jpg)
Live demo
• OpenCV (try it out!)
LKdemo
![Page 21: KLT tracker & triangulation Class 6 Read Shi and Tomasi’s paper on good features to track](https://reader033.vdocuments.net/reader033/viewer/2022052701/56649d775503460f94a592fe/html5/thumbnails/21.jpg)
Triangulation
C1m1
L1
m2
L2
M
C2
Triangulation
- calibration
- correspondences
![Page 22: KLT tracker & triangulation Class 6 Read Shi and Tomasi’s paper on good features to track](https://reader033.vdocuments.net/reader033/viewer/2022052701/56649d775503460f94a592fe/html5/thumbnails/22.jpg)
Triangulation• Backprojection
• Triangulation
Iterative least-squares
• Maximum Likelihood Triangulation
![Page 23: KLT tracker & triangulation Class 6 Read Shi and Tomasi’s paper on good features to track](https://reader033.vdocuments.net/reader033/viewer/2022052701/56649d775503460f94a592fe/html5/thumbnails/23.jpg)
Backprojection
• Represent point as intersection of row and column
Useful presentation for deriving and understanding multiple view geometry(notice 3D planes are linear in 2D point coordinates)
• Condition for solution?
![Page 24: KLT tracker & triangulation Class 6 Read Shi and Tomasi’s paper on good features to track](https://reader033.vdocuments.net/reader033/viewer/2022052701/56649d775503460f94a592fe/html5/thumbnails/24.jpg)
Next class: epipolar geometry