pyramidal implementation of lucas kanade feature tracker jia huang xiaoyan liu han xin yizhen tan
TRANSCRIPT
Pyramidal Implementation of Lucas Kanade Feature Tracker
Jia HuangXiaoyan Liu
Han XinYizhen Tan
Abstract
IntroductionTracking algorithm
Lucas-Kanade algorithm Iterative implementation
Tracking features analysis Feature lost Feature selection
Objective For a given point u in image A, find its
corresponding location v = u + d in image B.
Image A Image B
d
Residual function and Window size
2( ) ( , ) ( ( , ) ( , ))y yx x
x x x x
u wu w
x y x yx u w y u w
d d d A x y B x d y d
To find the location Minimize residual function:
: Integration window size
Small integration window Higher accuracy
Larger integration window Higher robustness
Nature tradeoff:
,x yw w
Pyramid Implementation of LK algorithm Calculate a set of pyramid representations of original image Apply traditional tracking algorithm for each level Results of current iteration is propagated to next iteration Key point: the same window size is used for each level
Top View Side View
Lucas-Kanade algorithm(1)
At the level L, we define images A and B: ( , ) [ 1, 1] [ 1, 1]x x x x y y y yx y p w p w p w p w
( , ) [ , ] [ , ]x x x x y y y yx y p w p w p w p w
( , ) ( , )LA x y I x y
( , ) ( , )L L Lx yB x y J x g y g
2( ) ( , ) ( ( , ) ( , ))y yx x
x x y y
p wp w
x y x yx p w y p w
v v v A x y B x v y v
Lucas-Kanade algorithm(2) At the optimum, the first derivative of
After first order Taylor expansion
Components in the equation above
( )| [0 0]
optv v
v
v
,
( )( ) ( ( , ) ( , ) )
y yx x
x x y y
p wp w
x p w y p w
v B B B Bv A x y B x y v
v x y x y
( , ) ( , ) ( , )I x y A x y B x y T
x
y
I B BI
I x y
2
2
1 ( )
2
y yx x
x x y y
T p wp wxx x y
x p w y p w yx y y
I II I Ivv
I II I Iv
Lucas-Kanade algorithm(3) Two derivative images are expressed:
With these notation, we can get:
The optimum optical flow vector is ( , ) ( 1, ) ( 1. )( , )
2x
A x y A x y A x yI x y
x
( , ) ( , 1) ( . 1)
( , )2y
A x y A x y A x yI x y
y
1optv G b
bG
Pyramidal diagram
Inner loop: K-level K initialized to 1, assume that the previous
computations from iterations 1,2,...,k-1 provide an initial guess
The new translated image according to
Iterative scheme of LK algorithm(1)
. , 3me g L
0mL
1 1 1[ ]k k k Tx yv v v
1kv
( , ) [ , ] [ , ]x x x x y y y yx y p w p w p w p w 1 1( , ) ( , )k k
k x yB x y B x v y v
Iterative scheme of LK algorithm(2) The goal: to compute the residual pixel motion vector
, that minimizes the error function
Image mismatch vector , where the image difference delta I k defined as:
New pixel displacement guess is computed for the next iteration step k+1:
[ ]k k kx y
2( ) ( , ) ( ( , ) ( , ))y yx x
x x y y
p wp wk k k kx y k x y
x p w y p w
A x y B x y
( , ) ( , )
( , ) ( , )
y yx x
x x y y
p wp wk x
k
x p w y p w k y
I x y I x yb
I x y I x y
( , ) ( , ) ( , )k kI x y A x y B x y
kbthk
1k k kv v
Iterative scheme of LK algorithm(3)
On average, 5 iterations are enough At the 1st iteration (k=1), the initial guess is set
to zero
The final solution for the optical flow vector is
Outer loop: L-levelThe vector d is propagated to the next level
L-1 and overall procedure is repeated L-1, L-2, …, 0
1
KL K k
k
v d v
0 [0 0]T
Declaring a Feature Lost
Several cases of lost feature the point falls outside of the image image patch around the tracked point varies
too much between image A and image B too large displacement
How to solve it combine a traditional tracking approach with an affine image matching
Feature Lost Example(1)
Image A Image B
Feature Lost Example(2)
Image A Image B
Feature Selection
Intuitive To select the point u on image A good to track.
Process steps: Compute the G matrix and λm
Call λmax the maximum value of λm Retain the pixels that have a λm value larger than a percentage of λmax Retain the local max. pixels Keep the subset of those pixels so that the minimum distance between pixels is larger than a threshold
Example of LK Feature Tracking
Image A Image B
More Examples
Image BImage A
Summary
Lucas-Kanade Feature Tracker is one of the most popular versions of two-frame differential methods for motion estimation
Iterative implementation of the Lucas-Kanade optical flow computation provides sufficient local tracking accuracy.
Thanks for your attention
Any question?