the klt (kanade-lucas-tomasi) tracker · 2018. 4. 18. · klt, finding best transformation p* how...
TRANSCRIPT
![Page 1: The KLT (Kanade-Lucas-Tomasi) tracker · 2018. 4. 18. · KLT, Finding Best Transformation p* How to find p* ? It would be possible to quantize the search space of p, enumerate all](https://reader035.vdocuments.net/reader035/viewer/2022081408/60588ef552ad6c6ca35a05ec/html5/thumbnails/1.jpg)
J. Matas, Tracking
The KLT
(Kanade-Lucas-Tomasi)
tracker
![Page 2: The KLT (Kanade-Lucas-Tomasi) tracker · 2018. 4. 18. · KLT, Finding Best Transformation p* How to find p* ? It would be possible to quantize the search space of p, enumerate all](https://reader035.vdocuments.net/reader035/viewer/2022081408/60588ef552ad6c6ca35a05ec/html5/thumbnails/2.jpg)
J. Matas, Tracking
KLT Introduction Given: • Input image I and the template image T • Region of interest (ROI) in T • Assumed transformation Wg from ROI in T to (part of) I :
Wg(x;gp):gR2p!jR2 (p = parameter) Find: • ap* such that T(x) is “close” to If(Wg(x;gp¤f)) for xj2jROI
Template image T Input image I
If(Wg(x;gp¤f)
2/14
![Page 3: The KLT (Kanade-Lucas-Tomasi) tracker · 2018. 4. 18. · KLT, Finding Best Transformation p* How to find p* ? It would be possible to quantize the search space of p, enumerate all](https://reader035.vdocuments.net/reader035/viewer/2022081408/60588ef552ad6c6ca35a05ec/html5/thumbnails/3.jpg)
J. Matas, Tracking
KLT, Transformation functions
Examples of transformation Wg(x;gp):gR2 c!jR2 (p = parameter)
Template image Tf(x),g xj=j(x1,x2j)T,
Input image Ig(y a), yk=j(y1,y2j)T.
• Translation by pk=j(p1,p2j)T , pj2jR2:
• Affine transformation, pj2jR2£3 :
• Rotation by p around origin, jp =jp 2j[0,g2¼a) :
3/14
![Page 4: The KLT (Kanade-Lucas-Tomasi) tracker · 2018. 4. 18. · KLT, Finding Best Transformation p* How to find p* ? It would be possible to quantize the search space of p, enumerate all](https://reader035.vdocuments.net/reader035/viewer/2022081408/60588ef552ad6c6ca35a05ec/html5/thumbnails/4.jpg)
J. Matas, Tracking
KLT, Measuring Fit
Measure of “closeness” of T(x) and If(Wg(x;gpf)), xj2jROI
SSD (used most often):
Optimization task: Find p* such that Note: In the subsequent text, for better clarity we use the following notation for ‘function at a point’ interchangeably:
4/14
![Page 5: The KLT (Kanade-Lucas-Tomasi) tracker · 2018. 4. 18. · KLT, Finding Best Transformation p* How to find p* ? It would be possible to quantize the search space of p, enumerate all](https://reader035.vdocuments.net/reader035/viewer/2022081408/60588ef552ad6c6ca35a05ec/html5/thumbnails/5.jpg)
J. Matas, Tracking
KLT, Finding Best Transformation p*
How to find p* ?
It would be possible to quantize the search space of p, enumerate all possibilities and take the one minimizing SSD. But this computationally expensive and would be infeasible in practice.
KLT approach:
1. Make the linearization (first-order Taylor approximation) inside brackets w.r.t. the parameter p
2. By this, the problem is converted to least-squares problem which is well understood and has a closed-form solution.
3. Because of approximation in step 1, iterate.
5/14
![Page 6: The KLT (Kanade-Lucas-Tomasi) tracker · 2018. 4. 18. · KLT, Finding Best Transformation p* How to find p* ? It would be possible to quantize the search space of p, enumerate all](https://reader035.vdocuments.net/reader035/viewer/2022081408/60588ef552ad6c6ca35a05ec/html5/thumbnails/6.jpg)
J. Matas, Tracking
Approximating the warp
Taylor approximation of If(Wg(x;gpf)) :
Jacobian of W image gradient im. for current p0
6/14
![Page 7: The KLT (Kanade-Lucas-Tomasi) tracker · 2018. 4. 18. · KLT, Finding Best Transformation p* How to find p* ? It would be possible to quantize the search space of p, enumerate all](https://reader035.vdocuments.net/reader035/viewer/2022081408/60588ef552ad6c6ca35a05ec/html5/thumbnails/7.jpg)
J. Matas, Tracking
Example, pure translation
Transformation:
Its Jacobian:
7/14
![Page 8: The KLT (Kanade-Lucas-Tomasi) tracker · 2018. 4. 18. · KLT, Finding Best Transformation p* How to find p* ? It would be possible to quantize the search space of p, enumerate all](https://reader035.vdocuments.net/reader035/viewer/2022081408/60588ef552ad6c6ca35a05ec/html5/thumbnails/8.jpg)
J. Matas, Tracking
Example, pure translation
image
gradient approximation
8/14
![Page 9: The KLT (Kanade-Lucas-Tomasi) tracker · 2018. 4. 18. · KLT, Finding Best Transformation p* How to find p* ? It would be possible to quantize the search space of p, enumerate all](https://reader035.vdocuments.net/reader035/viewer/2022081408/60588ef552ad6c6ca35a05ec/html5/thumbnails/9.jpg)
J. Matas, Tracking
LSQ problem to find Δp*
Solve for Δp* :
Taylor approximation of If(Wg(x;gpf)) :
9/14
![Page 10: The KLT (Kanade-Lucas-Tomasi) tracker · 2018. 4. 18. · KLT, Finding Best Transformation p* How to find p* ? It would be possible to quantize the search space of p, enumerate all](https://reader035.vdocuments.net/reader035/viewer/2022081408/60588ef552ad6c6ca35a05ec/html5/thumbnails/10.jpg)
J. Matas, Tracking
KLT, Finding Best Transformation p*
This:
has a form of a classical least-squares problem:
where
• rows of matrix A are equal to rIfT(@aWg=@gp) (for a given x and p0),
• z = Δp , and
• elements of b equal to T(x)i¡iI(Wg(x;gp0)), that is, current residuals for given x.
Such problem has a well-known solution, zj=j(ATA)¡1ATjb.
10/14
![Page 11: The KLT (Kanade-Lucas-Tomasi) tracker · 2018. 4. 18. · KLT, Finding Best Transformation p* How to find p* ? It would be possible to quantize the search space of p, enumerate all](https://reader035.vdocuments.net/reader035/viewer/2022081408/60588ef552ad6c6ca35a05ec/html5/thumbnails/11.jpg)
J. Matas, Tracking
Example, pure translation
Transformation:
Its Jacobian:
Solution for update (from previous slide):
Rows of A:
Stability of computation will rely on rank of this matrix
11/14
![Page 12: The KLT (Kanade-Lucas-Tomasi) tracker · 2018. 4. 18. · KLT, Finding Best Transformation p* How to find p* ? It would be possible to quantize the search space of p, enumerate all](https://reader035.vdocuments.net/reader035/viewer/2022081408/60588ef552ad6c6ca35a05ec/html5/thumbnails/12.jpg)
J. Matas, Tracking
Example, pure translation
The stability of computation will depend on invertibility of this matrix.
We have already encountered this matrix in this course.
It is directly related to Harris corner detector.
For obvious reasons, the KLT tracking for translation is well posed when
the ROI contains a corner, as opposed to a flat region or an edge.
Aperture problem:
(image courtesy of Wolfe et al. Sensation & Perception)
12/14
![Page 13: The KLT (Kanade-Lucas-Tomasi) tracker · 2018. 4. 18. · KLT, Finding Best Transformation p* How to find p* ? It would be possible to quantize the search space of p, enumerate all](https://reader035.vdocuments.net/reader035/viewer/2022081408/60588ef552ad6c6ca35a05ec/html5/thumbnails/13.jpg)
J. Matas, Tracking
KLT, algorithm summary
1. Input: current estimate p0 . Solve for Δp* :
2. Update, p0 ← p0 + Δp*
3. If not converged, go to 1.
4. Best transformation p* = p0.
Taylor approximation of If(Wg(x;gpf)) :
13/14
![Page 14: The KLT (Kanade-Lucas-Tomasi) tracker · 2018. 4. 18. · KLT, Finding Best Transformation p* How to find p* ? It would be possible to quantize the search space of p, enumerate all](https://reader035.vdocuments.net/reader035/viewer/2022081408/60588ef552ad6c6ca35a05ec/html5/thumbnails/14.jpg)
J. Matas, Tracking
Multi-resolution Lucas-Kanade
KLT is based on 1st order Taylor approximation and cannot find the solution if the displacement is too high compared to frequencies in the image.
Possible solution is to blur the image on the scale of assumed displacement and get a rough estimate.
Subsequently, continue on less blurred image and make an update to that estimate. Repeat.
This is efficiently implemented using image pyramids.
slide credit:
Patrick Perez
14/14